[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250109 20:57:50.605443 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:34671,127.17.14.189:34769,127.17.14.188:45963
I20250109 20:57:50.607056 17466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:57:50.607962 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:57:50.623642 17475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:50.624049 17472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:50.625859 17466 server_base.cc:1034] running on GCE node
W20250109 20:57:50.624472 17473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:51.781924 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:57:51.782150 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:57:51.782299 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456271782284 us; error 0 us; skew 500 ppm
I20250109 20:57:51.783046 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:57:51.788172 17466 webserver.cc:458] Webserver started at http://127.17.14.190:34177/ using document root <none> and password file <none>
I20250109 20:57:51.788906 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:57:51.789108 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:57:51.789462 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:57:51.795420 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "2552a73bef9e45a0bc674029ae573cdc"
format_stamp: "Formatted at 2025-01-09 20:57:51 on dist-test-slave-j54z"
I20250109 20:57:51.802711 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250109 20:57:51.807435 17481 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:51.808413 17466 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250109 20:57:51.808696 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "2552a73bef9e45a0bc674029ae573cdc"
format_stamp: "Formatted at 2025-01-09 20:57:51 on dist-test-slave-j54z"
I20250109 20:57:51.808961 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:57:51.866950 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:57:51.868284 17466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:57:51.868640 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:57:51.936271 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:34671
I20250109 20:57:51.936342 17532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:34671 every 8 connection(s)
I20250109 20:57:51.942139 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 20:57:51.943116 17533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:57:51.949131 17535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:51.950456 17536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:51.954177 17538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:51.955013 17466 server_base.cc:1034] running on GCE node
I20250109 20:57:51.955857 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:57:51.956135 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:57:51.956339 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456271956322 us; error 0 us; skew 500 ppm
I20250109 20:57:51.957083 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:57:51.961269 17466 webserver.cc:458] Webserver started at http://127.17.14.189:43613/ using document root <none> and password file <none>
I20250109 20:57:51.961964 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:57:51.962219 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:57:51.962565 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:57:51.964035 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-1-root/instance:
uuid: "5370928dcee047f4be450ed478f6a26f"
format_stamp: "Formatted at 2025-01-09 20:57:51 on dist-test-slave-j54z"
I20250109 20:57:51.970638 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250109 20:57:51.968935 17533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:57:51.977017 17543 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:51.978070 17466 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20250109 20:57:51.978413 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-1-root
uuid: "5370928dcee047f4be450ed478f6a26f"
format_stamp: "Formatted at 2025-01-09 20:57:51 on dist-test-slave-j54z"
I20250109 20:57:51.978699 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:57:51.996430 17533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:57:51.998713 17485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.14.189:34769: connect: Connection refused (error 111)
W20250109 20:57:52.002863 17533 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.14.189:34769: Network error: Client connection negotiation failed: client connection to 127.17.14.189:34769: connect: Connection refused (error 111)
I20250109 20:57:52.004364 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:57:52.005609 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:57:52.052335 17533 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } attempt: 1
W20250109 20:57:52.056485 17533 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.14.189:34769: Network error: Client connection negotiation failed: client connection to 127.17.14.189:34769: connect: Connection refused (error 111)
I20250109 20:57:52.070631 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.189:34769
I20250109 20:57:52.070796 17597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.189:34769 every 8 connection(s)
I20250109 20:57:52.075179 17598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:57:52.076153 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 20:57:52.081562 17598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:57:52.082561 17600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:52.085000 17601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:52.089761 17604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:52.090847 17466 server_base.cc:1034] running on GCE node
I20250109 20:57:52.091531 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:57:52.091740 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:57:52.091914 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456272091896 us; error 0 us; skew 500 ppm
I20250109 20:57:52.092427 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:57:52.095988 17466 webserver.cc:458] Webserver started at http://127.17.14.188:42865/ using document root <none> and password file <none>
I20250109 20:57:52.096553 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:57:52.096763 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:57:52.096596 17598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:57:52.097030 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:57:52.098373 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-2-root/instance:
uuid: "4bb8f00c576b46abaddaf5ff786da3e2"
format_stamp: "Formatted at 2025-01-09 20:57:52 on dist-test-slave-j54z"
I20250109 20:57:52.103796 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20250109 20:57:52.108886 17610 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:52.109655 17466 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250109 20:57:52.109915 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-2-root
uuid: "4bb8f00c576b46abaddaf5ff786da3e2"
format_stamp: "Formatted at 2025-01-09 20:57:52 on dist-test-slave-j54z"
I20250109 20:57:52.110172 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:57:52.113997 17598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:57:52.119798 17598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.14.188:45963: Network error: Client connection negotiation failed: client connection to 127.17.14.188:45963: connect: Connection refused (error 111)
I20250109 20:57:52.125013 17533 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } attempt: 2
I20250109 20:57:52.133536 17533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:57:52.136809 17533 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.14.188:45963: Network error: Client connection negotiation failed: client connection to 127.17.14.188:45963: connect: Connection refused (error 111)
I20250109 20:57:52.142016 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:57:52.143232 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:57:52.163190 17598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } attempt: 1
W20250109 20:57:52.166990 17598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.14.188:45963: Network error: Client connection negotiation failed: client connection to 127.17.14.188:45963: connect: Connection refused (error 111)
I20250109 20:57:52.168147 17533 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } attempt: 1
W20250109 20:57:52.173081 17533 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.14.188:45963: Network error: Client connection negotiation failed: client connection to 127.17.14.188:45963: connect: Connection refused (error 111)
I20250109 20:57:52.208326 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.188:45963
I20250109 20:57:52.208406 17662 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.188:45963 every 8 connection(s)
I20250109 20:57:52.212602 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 20:57:52.213579 17663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:57:52.218688 17663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:57:52.229734 17663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:57:52.237200 17663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:57:52.240506 17533 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } attempt: 2
I20250109 20:57:52.242444 17598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } attempt: 2
I20250109 20:57:52.260170 17598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f: Bootstrap starting.
I20250109 20:57:52.260200 17533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc: Bootstrap starting.
I20250109 20:57:52.260180 17663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2: Bootstrap starting.
I20250109 20:57:52.266523 17598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f: Neither blocks nor log segments found. Creating new log.
I20250109 20:57:52.266607 17663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2: Neither blocks nor log segments found. Creating new log.
I20250109 20:57:52.267181 17533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc: Neither blocks nor log segments found. Creating new log.
I20250109 20:57:52.268549 17598 log.cc:826] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f: Log is configured to *not* fsync() on all Append() calls
I20250109 20:57:52.275414 17663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2: No bootstrap required, opened a new log
I20250109 20:57:52.275517 17598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f: No bootstrap required, opened a new log
I20250109 20:57:52.276166 17533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc: No bootstrap required, opened a new log
I20250109 20:57:52.294692 17598 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } }
I20250109 20:57:52.294610 17533 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } }
I20250109 20:57:52.294610 17663 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } }
I20250109 20:57:52.295435 17598 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:57:52.295624 17533 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:57:52.295667 17663 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:57:52.295778 17598 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5370928dcee047f4be450ed478f6a26f, State: Initialized, Role: FOLLOWER
I20250109 20:57:52.296020 17533 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2552a73bef9e45a0bc674029ae573cdc, State: Initialized, Role: FOLLOWER
I20250109 20:57:52.296092 17663 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb8f00c576b46abaddaf5ff786da3e2, State: Initialized, Role: FOLLOWER
I20250109 20:57:52.296757 17598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } }
I20250109 20:57:52.296788 17533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } }
I20250109 20:57:52.296864 17663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } }
I20250109 20:57:52.301564 17674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } } }
I20250109 20:57:52.302510 17674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:57:52.302349 17673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } } }
I20250109 20:57:52.303172 17673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:57:52.303520 17663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:57:52.301558 17672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } } }
I20250109 20:57:52.304430 17672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:57:52.304960 17533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:57:52.303520 17598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [sys.catalog]: configured and running, proceeding with master startup.
W20250109 20:57:52.356660 17703 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 20:57:52.357167 17703 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 20:57:52.358078 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250109 20:57:52.358350 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250109 20:57:52.358994 17706 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 20:57:52.359246 17706 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 20:57:52.360229 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:57:52.370388 17709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:52.370455 17707 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 20:57:52.370891 17707 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250109 20:57:52.373406 17487 tablet.cc:2367] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 20:57:52.375974 17710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:52.379736 17466 server_base.cc:1034] running on GCE node
W20250109 20:57:52.380442 17712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:52.381330 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:57:52.381570 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:57:52.381726 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456272381706 us; error 0 us; skew 500 ppm
I20250109 20:57:52.382184 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:57:52.384774 17466 webserver.cc:458] Webserver started at http://127.17.14.129:35779/ using document root <none> and password file <none>
I20250109 20:57:52.385217 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:57:52.385392 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:57:52.385665 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:57:52.386685 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-0-root/instance:
uuid: "e4a675eb3dba403a8cd3f725f30a4c37"
format_stamp: "Formatted at 2025-01-09 20:57:52 on dist-test-slave-j54z"
I20250109 20:57:52.391278 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20250109 20:57:52.394662 17717 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:52.395437 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 20:57:52.395677 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "e4a675eb3dba403a8cd3f725f30a4c37"
format_stamp: "Formatted at 2025-01-09 20:57:52 on dist-test-slave-j54z"
I20250109 20:57:52.395924 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:57:52.456543 17674 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:57:52.457120 17674 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } }
I20250109 20:57:52.460783 17674 leader_election.cc:290] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2552a73bef9e45a0bc674029ae573cdc (127.17.14.190:34671), 5370928dcee047f4be450ed478f6a26f (127.17.14.189:34769)
I20250109 20:57:52.467432 17508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2552a73bef9e45a0bc674029ae573cdc" is_pre_election: true
I20250109 20:57:52.468369 17508 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bb8f00c576b46abaddaf5ff786da3e2 in term 0.
I20250109 20:57:52.469854 17614 leader_election.cc:304] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2552a73bef9e45a0bc674029ae573cdc, 4bb8f00c576b46abaddaf5ff786da3e2; no voters:
I20250109 20:57:52.470559 17674 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:57:52.470933 17674 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:57:52.471400 17674 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:57:52.473812 17573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5370928dcee047f4be450ed478f6a26f" is_pre_election: true
I20250109 20:57:52.475101 17573 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bb8f00c576b46abaddaf5ff786da3e2 in term 0.
I20250109 20:57:52.481374 17674 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } }
I20250109 20:57:52.483533 17674 leader_election.cc:290] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [CANDIDATE]: Term 1 election: Requested vote from peers 2552a73bef9e45a0bc674029ae573cdc (127.17.14.190:34671), 5370928dcee047f4be450ed478f6a26f (127.17.14.189:34769)
I20250109 20:57:52.508780 17508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2552a73bef9e45a0bc674029ae573cdc"
I20250109 20:57:52.511210 17508 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:57:52.513789 17573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5370928dcee047f4be450ed478f6a26f"
I20250109 20:57:52.514430 17573 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:57:52.518658 17508 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bb8f00c576b46abaddaf5ff786da3e2 in term 1.
I20250109 20:57:52.519934 17614 leader_election.cc:304] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2552a73bef9e45a0bc674029ae573cdc, 4bb8f00c576b46abaddaf5ff786da3e2; no voters:
I20250109 20:57:52.520891 17573 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bb8f00c576b46abaddaf5ff786da3e2 in term 1.
I20250109 20:57:52.531432 17674 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:57:52.533630 17674 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 1 LEADER]: Becoming Leader. State: Replica: 4bb8f00c576b46abaddaf5ff786da3e2, State: Running, Role: LEADER
I20250109 20:57:52.534638 17674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } }
I20250109 20:57:52.543186 17722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb8f00c576b46abaddaf5ff786da3e2. Latest consensus state: current_term: 1 leader_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } } }
I20250109 20:57:52.544188 17722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [sys.catalog]: This master's current role is: LEADER
I20250109 20:57:52.547278 17724 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:57:52.549137 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:57:52.550468 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:57:52.553076 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:57:52.555341 17724 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:57:52.557512 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:57:52.557905 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250109 20:57:52.558182 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:57:52.558343 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:52.573702 17573 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [term 1 FOLLOWER]: Refusing update from remote peer 4bb8f00c576b46abaddaf5ff786da3e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:57:52.575405 17508 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [term 1 FOLLOWER]: Refusing update from remote peer 4bb8f00c576b46abaddaf5ff786da3e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:57:52.576843 17722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:57:52.578083 17674 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250109 20:57:52.609763 17673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb8f00c576b46abaddaf5ff786da3e2. Latest consensus state: current_term: 1 leader_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } } }
I20250109 20:57:52.610592 17673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:57:52.610462 17672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb8f00c576b46abaddaf5ff786da3e2. Latest consensus state: current_term: 1 leader_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } } }
I20250109 20:57:52.611212 17672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:57:52.617453 17728 mvcc.cc:204] Tried to move back new op lower bound from 7112524892442562560 to 7112524892312518656. Current Snapshot: MvccSnapshot[applied={T|T < 7112524892442562560}]
I20250109 20:57:52.618141 17674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } } }
I20250109 20:57:52.619306 17674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [sys.catalog]: This master's current role is: LEADER
I20250109 20:57:52.620702 17724 catalog_manager.cc:1348] Generated new cluster ID: 1457b1c6dc464745984aec01395b6035
I20250109 20:57:52.621004 17724 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:57:52.632506 17672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } } }
I20250109 20:57:52.633289 17672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:57:52.655573 17674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } } }
I20250109 20:57:52.657419 17674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [sys.catalog]: This master's current role is: LEADER
I20250109 20:57:52.670248 17724 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 20:57:52.672472 17724 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:57:52.676497 17673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2552a73bef9e45a0bc674029ae573cdc" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 34671 } } peers { permanent_uuid: "5370928dcee047f4be450ed478f6a26f" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 34769 } } peers { permanent_uuid: "4bb8f00c576b46abaddaf5ff786da3e2" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 45963 } } }
I20250109 20:57:52.677233 17673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:57:52.701639 17724 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2: Generated new TSK 0
I20250109 20:57:52.702785 17724 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:57:52.769116 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:35493
I20250109 20:57:52.769342 17791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:35493 every 8 connection(s)
I20250109 20:57:52.779351 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:57:52.799062 17798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:52.811560 17800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:52.812407 17804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:52.813588 17466 server_base.cc:1034] running on GCE node
I20250109 20:57:52.814838 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:57:52.815100 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:57:52.815303 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456272815287 us; error 0 us; skew 500 ppm
I20250109 20:57:52.815843 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:57:52.816097 17792 heartbeater.cc:346] Connected to a master server at 127.17.14.188:45963
I20250109 20:57:52.816483 17792 heartbeater.cc:463] Registering TS with master...
I20250109 20:57:52.817773 17792 heartbeater.cc:510] Master 127.17.14.188:45963 requested a full tablet report, sending...
I20250109 20:57:52.821761 17628 ts_manager.cc:194] Registered new tserver with Master: e4a675eb3dba403a8cd3f725f30a4c37 (127.17.14.129:35493)
I20250109 20:57:52.825222 17628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39406
I20250109 20:57:52.830080 17793 heartbeater.cc:346] Connected to a master server at 127.17.14.189:34769
I20250109 20:57:52.830521 17793 heartbeater.cc:463] Registering TS with master...
I20250109 20:57:52.831405 17793 heartbeater.cc:510] Master 127.17.14.189:34769 requested a full tablet report, sending...
I20250109 20:57:52.835541 17563 ts_manager.cc:194] Registered new tserver with Master: e4a675eb3dba403a8cd3f725f30a4c37 (127.17.14.129:35493)
I20250109 20:57:52.838871 17795 heartbeater.cc:346] Connected to a master server at 127.17.14.190:34671
I20250109 20:57:52.839324 17795 heartbeater.cc:463] Registering TS with master...
I20250109 20:57:52.839444 17466 webserver.cc:458] Webserver started at http://127.17.14.130:44667/ using document root <none> and password file <none>
I20250109 20:57:52.840031 17795 heartbeater.cc:510] Master 127.17.14.190:34671 requested a full tablet report, sending...
I20250109 20:57:52.840171 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:57:52.840498 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:57:52.840852 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:57:52.842115 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-1-root/instance:
uuid: "4a59e879d70845a79c27bcad5e0abf6a"
format_stamp: "Formatted at 2025-01-09 20:57:52 on dist-test-slave-j54z"
I20250109 20:57:52.843031 17498 ts_manager.cc:194] Registered new tserver with Master: e4a675eb3dba403a8cd3f725f30a4c37 (127.17.14.129:35493)
I20250109 20:57:52.846904 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250109 20:57:52.850450 17809 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:52.851286 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:57:52.851732 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "4a59e879d70845a79c27bcad5e0abf6a"
format_stamp: "Formatted at 2025-01-09 20:57:52 on dist-test-slave-j54z"
I20250109 20:57:52.852211 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:57:52.865835 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:57:52.867043 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:57:52.868460 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:57:52.871928 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:57:52.872116 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:52.872346 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:57:52.872483 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:52.971210 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:41155
I20250109 20:57:52.971437 17871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:41155 every 8 connection(s)
I20250109 20:57:52.997395 17872 heartbeater.cc:346] Connected to a master server at 127.17.14.188:45963
I20250109 20:57:52.997836 17872 heartbeater.cc:463] Registering TS with master...
I20250109 20:57:52.998829 17872 heartbeater.cc:510] Master 127.17.14.188:45963 requested a full tablet report, sending...
I20250109 20:57:52.999673 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 20:57:53.001366 17628 ts_manager.cc:194] Registered new tserver with Master: 4a59e879d70845a79c27bcad5e0abf6a (127.17.14.130:41155)
I20250109 20:57:53.003134 17628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39418
I20250109 20:57:53.016026 17873 heartbeater.cc:346] Connected to a master server at 127.17.14.189:34769
I20250109 20:57:53.016396 17873 heartbeater.cc:463] Registering TS with master...
I20250109 20:57:53.017180 17873 heartbeater.cc:510] Master 127.17.14.189:34769 requested a full tablet report, sending...
I20250109 20:57:53.018138 17875 heartbeater.cc:346] Connected to a master server at 127.17.14.190:34671
I20250109 20:57:53.018651 17875 heartbeater.cc:463] Registering TS with master...
I20250109 20:57:53.019726 17875 heartbeater.cc:510] Master 127.17.14.190:34671 requested a full tablet report, sending...
I20250109 20:57:53.023295 17563 ts_manager.cc:194] Registered new tserver with Master: 4a59e879d70845a79c27bcad5e0abf6a (127.17.14.130:41155)
W20250109 20:57:53.374534 17878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:53.375440 17879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:53.463774 17703 catalog_manager.cc:1260] Loaded cluster ID: 1457b1c6dc464745984aec01395b6035
I20250109 20:57:53.464062 17703 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f: loading cluster ID for follower catalog manager: success
I20250109 20:57:53.471568 17703 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f: acquiring CA information for follower catalog manager: success
I20250109 20:57:53.476985 17703 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:57:53.493250 17498 ts_manager.cc:194] Registered new tserver with Master: 4a59e879d70845a79c27bcad5e0abf6a (127.17.14.130:41155)
I20250109 20:57:53.508563 17707 catalog_manager.cc:1260] Loaded cluster ID: 1457b1c6dc464745984aec01395b6035
I20250109 20:57:53.508873 17707 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc: loading cluster ID for follower catalog manager: success
W20250109 20:57:53.522616 17594 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0060 after lost signal to thread 17469
I20250109 20:57:53.534584 17707 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc: acquiring CA information for follower catalog manager: success
W20250109 20:57:53.556967 17882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:53.541299 17707 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:57:53.562691 17466 server_base.cc:1034] running on GCE node
I20250109 20:57:53.564954 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:57:53.565333 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:57:53.565670 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456273565648 us; error 0 us; skew 500 ppm
I20250109 20:57:53.566491 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:57:53.584704 17466 webserver.cc:458] Webserver started at http://127.17.14.131:38937/ using document root <none> and password file <none>
I20250109 20:57:53.585546 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:57:53.585860 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:57:53.586268 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:57:53.591372 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-2-root/instance:
uuid: "1fa1800445b04f909f6aaf0141308e76"
format_stamp: "Formatted at 2025-01-09 20:57:53 on dist-test-slave-j54z"
I20250109 20:57:53.603976 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.007s sys 0.003s
I20250109 20:57:53.623327 17887 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:53.624389 17466 fs_manager.cc:730] Time spent opening block manager: real 0.018s user 0.001s sys 0.015s
I20250109 20:57:53.624774 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "1fa1800445b04f909f6aaf0141308e76"
format_stamp: "Formatted at 2025-01-09 20:57:53 on dist-test-slave-j54z"
I20250109 20:57:53.625154 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:57:53.681370 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:57:53.683145 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:57:53.691596 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:57:53.702219 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:57:53.702587 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:53.702921 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:57:53.703181 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:53.829869 17792 heartbeater.cc:502] Master 127.17.14.188:45963 was elected leader, sending a full tablet report...
I20250109 20:57:54.003409 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:36145
I20250109 20:57:54.003508 17949 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:36145 every 8 connection(s)
I20250109 20:57:54.008239 17872 heartbeater.cc:502] Master 127.17.14.188:45963 was elected leader, sending a full tablet report...
I20250109 20:57:54.080245 17950 heartbeater.cc:346] Connected to a master server at 127.17.14.188:45963
I20250109 20:57:54.080704 17950 heartbeater.cc:463] Registering TS with master...
I20250109 20:57:54.081669 17950 heartbeater.cc:510] Master 127.17.14.188:45963 requested a full tablet report, sending...
I20250109 20:57:54.083956 17628 ts_manager.cc:194] Registered new tserver with Master: 1fa1800445b04f909f6aaf0141308e76 (127.17.14.131:36145)
I20250109 20:57:54.084239 17951 heartbeater.cc:346] Connected to a master server at 127.17.14.189:34769
I20250109 20:57:54.084650 17951 heartbeater.cc:463] Registering TS with master...
I20250109 20:57:54.085698 17628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39422
I20250109 20:57:54.085831 17951 heartbeater.cc:510] Master 127.17.14.189:34769 requested a full tablet report, sending...
I20250109 20:57:54.088461 17563 ts_manager.cc:194] Registered new tserver with Master: 1fa1800445b04f909f6aaf0141308e76 (127.17.14.131:36145)
I20250109 20:57:54.091653 17954 heartbeater.cc:346] Connected to a master server at 127.17.14.190:34671
I20250109 20:57:54.092005 17954 heartbeater.cc:463] Registering TS with master...
I20250109 20:57:54.092639 17954 heartbeater.cc:510] Master 127.17.14.190:34671 requested a full tablet report, sending...
I20250109 20:57:54.095263 17498 ts_manager.cc:194] Registered new tserver with Master: 1fa1800445b04f909f6aaf0141308e76 (127.17.14.131:36145)
I20250109 20:57:54.096443 17466 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.034678567s
I20250109 20:57:54.348954 17701 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:57:54.374869 17466 test_util.cc:274] Using random seed: 996602037
I20250109 20:57:54.469990 17628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39430:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:57:54.545900 17868 debug-util.cc:398] Leaking SignalData structure 0x7b08001cd060 after lost signal to thread 17469
W20250109 20:57:54.548979 17868 debug-util.cc:398] Leaking SignalData structure 0x7b08001da300 after lost signal to thread 17791
W20250109 20:57:54.552755 17868 debug-util.cc:398] Leaking SignalData structure 0x7b08001dac40 after lost signal to thread 17871
I20250109 20:57:54.650059 17757 tablet_service.cc:1467] Processing CreateTablet for tablet c7e0c6e2f8a54737887da84813c6faee (DEFAULT_TABLE table=test-workload [id=7c764ba3c3084e7d8610deb25ca680a9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250109 20:57:54.650893 17756 tablet_service.cc:1467] Processing CreateTablet for tablet f2529fbe4ada45e08d322e0e1dfe2426 (DEFAULT_TABLE table=test-workload [id=7c764ba3c3084e7d8610deb25ca680a9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 20:57:54.652120 17757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7e0c6e2f8a54737887da84813c6faee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:57:54.652793 17756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2529fbe4ada45e08d322e0e1dfe2426. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:57:54.673933 17837 tablet_service.cc:1467] Processing CreateTablet for tablet 25e059e377a646ad934df7702012572b (DEFAULT_TABLE table=test-workload [id=7c764ba3c3084e7d8610deb25ca680a9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 20:57:54.675470 17837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25e059e377a646ad934df7702012572b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:57:54.683456 17915 tablet_service.cc:1467] Processing CreateTablet for tablet da37c46e01cb413da83de231936bc15b (DEFAULT_TABLE table=test-workload [id=7c764ba3c3084e7d8610deb25ca680a9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 20:57:54.684923 17915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da37c46e01cb413da83de231936bc15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:57:54.686564 17984 tablet_bootstrap.cc:492] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37: Bootstrap starting.
I20250109 20:57:54.706947 17985 tablet_bootstrap.cc:492] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76: Bootstrap starting.
I20250109 20:57:54.713793 17984 tablet_bootstrap.cc:654] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37: Neither blocks nor log segments found. Creating new log.
I20250109 20:57:54.721514 17985 tablet_bootstrap.cc:654] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76: Neither blocks nor log segments found. Creating new log.
I20250109 20:57:54.741145 17984 tablet_bootstrap.cc:492] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37: No bootstrap required, opened a new log
I20250109 20:57:54.741703 17986 tablet_bootstrap.cc:492] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a: Bootstrap starting.
I20250109 20:57:54.743221 17984 ts_tablet_manager.cc:1397] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37: Time spent bootstrapping tablet: real 0.057s user 0.017s sys 0.005s
I20250109 20:57:54.748052 17985 tablet_bootstrap.cc:492] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76: No bootstrap required, opened a new log
I20250109 20:57:54.751346 17984 raft_consensus.cc:357] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a675eb3dba403a8cd3f725f30a4c37" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35493 } }
I20250109 20:57:54.751976 17984 raft_consensus.cc:383] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:57:54.752285 17984 raft_consensus.cc:738] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a675eb3dba403a8cd3f725f30a4c37, State: Initialized, Role: FOLLOWER
I20250109 20:57:54.752947 17984 consensus_queue.cc:260] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a675eb3dba403a8cd3f725f30a4c37" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35493 } }
I20250109 20:57:54.753619 17984 raft_consensus.cc:397] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:57:54.753927 17984 raft_consensus.cc:491] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:57:54.754251 17984 raft_consensus.cc:3054] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:57:54.756311 17985 ts_tablet_manager.cc:1397] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.011s
I20250109 20:57:54.762471 17984 raft_consensus.cc:513] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a675eb3dba403a8cd3f725f30a4c37" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35493 } }
I20250109 20:57:54.763698 17984 leader_election.cc:304] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4a675eb3dba403a8cd3f725f30a4c37; no voters:
I20250109 20:57:54.769179 17986 tablet_bootstrap.cc:654] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a: Neither blocks nor log segments found. Creating new log.
I20250109 20:57:54.769677 17985 raft_consensus.cc:357] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa1800445b04f909f6aaf0141308e76" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 36145 } }
I20250109 20:57:54.770445 17985 raft_consensus.cc:383] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:57:54.770834 17985 raft_consensus.cc:738] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa1800445b04f909f6aaf0141308e76, State: Initialized, Role: FOLLOWER
I20250109 20:57:54.771757 17985 consensus_queue.cc:260] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa1800445b04f909f6aaf0141308e76" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 36145 } }
I20250109 20:57:54.772677 17985 raft_consensus.cc:397] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:57:54.773106 17985 raft_consensus.cc:491] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:57:54.773530 17985 raft_consensus.cc:3054] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:57:54.784808 17985 raft_consensus.cc:513] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa1800445b04f909f6aaf0141308e76" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 36145 } }
I20250109 20:57:54.786302 17985 leader_election.cc:304] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fa1800445b04f909f6aaf0141308e76; no voters:
I20250109 20:57:54.788457 17986 tablet_bootstrap.cc:492] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a: No bootstrap required, opened a new log
I20250109 20:57:54.788937 17986 ts_tablet_manager.cc:1397] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.006s
I20250109 20:57:54.791662 17986 raft_consensus.cc:357] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a59e879d70845a79c27bcad5e0abf6a" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 41155 } }
I20250109 20:57:54.792197 17986 raft_consensus.cc:383] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:57:54.792513 17986 raft_consensus.cc:738] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a59e879d70845a79c27bcad5e0abf6a, State: Initialized, Role: FOLLOWER
I20250109 20:57:54.793152 17986 consensus_queue.cc:260] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a59e879d70845a79c27bcad5e0abf6a" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 41155 } }
I20250109 20:57:54.793922 17985 leader_election.cc:290] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:57:54.794298 17986 raft_consensus.cc:397] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:57:54.795975 17986 raft_consensus.cc:491] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:57:54.796332 17986 raft_consensus.cc:3054] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:57:54.796413 17991 raft_consensus.cc:2798] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:57:54.800148 17989 raft_consensus.cc:2798] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:57:54.801569 17984 leader_election.cc:290] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:57:54.807041 17985 ts_tablet_manager.cc:1428] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76: Time spent starting tablet: real 0.048s user 0.011s sys 0.007s
I20250109 20:57:54.807682 17950 heartbeater.cc:502] Master 127.17.14.188:45963 was elected leader, sending a full tablet report...
I20250109 20:57:54.810467 17991 raft_consensus.cc:695] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa1800445b04f909f6aaf0141308e76, State: Running, Role: LEADER
I20250109 20:57:54.811208 17986 raft_consensus.cc:513] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a59e879d70845a79c27bcad5e0abf6a" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 41155 } }
I20250109 20:57:54.812124 17986 leader_election.cc:304] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a59e879d70845a79c27bcad5e0abf6a; no voters:
I20250109 20:57:54.812088 17991 consensus_queue.cc:237] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa1800445b04f909f6aaf0141308e76" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 36145 } }
I20250109 20:57:54.822546 17996 raft_consensus.cc:2798] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:57:54.822607 17984 ts_tablet_manager.cc:1428] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37: Time spent starting tablet: real 0.079s user 0.020s sys 0.006s
I20250109 20:57:54.823671 17986 leader_election.cc:290] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:57:54.825027 17984 tablet_bootstrap.cc:492] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37: Bootstrap starting.
I20250109 20:57:54.830179 17989 raft_consensus.cc:695] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [term 1 LEADER]: Becoming Leader. State: Replica: e4a675eb3dba403a8cd3f725f30a4c37, State: Running, Role: LEADER
I20250109 20:57:54.831561 17989 consensus_queue.cc:237] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a675eb3dba403a8cd3f725f30a4c37" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35493 } }
I20250109 20:57:54.866701 17984 tablet_bootstrap.cc:654] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37: Neither blocks nor log segments found. Creating new log.
I20250109 20:57:54.864298 17627 catalog_manager.cc:5526] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4a675eb3dba403a8cd3f725f30a4c37 (127.17.14.129). New cstate: current_term: 1 leader_uuid: "e4a675eb3dba403a8cd3f725f30a4c37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a675eb3dba403a8cd3f725f30a4c37" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35493 } health_report { overall_health: HEALTHY } } }
I20250109 20:57:54.868423 17986 ts_tablet_manager.cc:1428] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a: Time spent starting tablet: real 0.079s user 0.020s sys 0.024s
I20250109 20:57:54.863555 17628 catalog_manager.cc:5526] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fa1800445b04f909f6aaf0141308e76 (127.17.14.131). New cstate: current_term: 1 leader_uuid: "1fa1800445b04f909f6aaf0141308e76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa1800445b04f909f6aaf0141308e76" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 36145 } health_report { overall_health: HEALTHY } } }
I20250109 20:57:54.947166 17996 raft_consensus.cc:695] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [term 1 LEADER]: Becoming Leader. State: Replica: 4a59e879d70845a79c27bcad5e0abf6a, State: Running, Role: LEADER
I20250109 20:57:54.959383 17996 consensus_queue.cc:237] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a59e879d70845a79c27bcad5e0abf6a" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 41155 } }
I20250109 20:57:54.989336 17628 catalog_manager.cc:5526] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a59e879d70845a79c27bcad5e0abf6a (127.17.14.130). New cstate: current_term: 1 leader_uuid: "4a59e879d70845a79c27bcad5e0abf6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a59e879d70845a79c27bcad5e0abf6a" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 41155 } health_report { overall_health: HEALTHY } } }
I20250109 20:57:54.990748 17984 tablet_bootstrap.cc:492] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37: No bootstrap required, opened a new log
I20250109 20:57:54.991255 17984 ts_tablet_manager.cc:1397] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37: Time spent bootstrapping tablet: real 0.166s user 0.010s sys 0.005s
I20250109 20:57:54.994027 17984 raft_consensus.cc:357] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a675eb3dba403a8cd3f725f30a4c37" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35493 } }
I20250109 20:57:54.994657 17984 raft_consensus.cc:383] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:57:54.994993 17984 raft_consensus.cc:738] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a675eb3dba403a8cd3f725f30a4c37, State: Initialized, Role: FOLLOWER
I20250109 20:57:54.995693 17984 consensus_queue.cc:260] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a675eb3dba403a8cd3f725f30a4c37" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35493 } }
I20250109 20:57:55.002142 17984 raft_consensus.cc:397] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:57:55.002525 17984 raft_consensus.cc:491] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:57:55.003041 17984 raft_consensus.cc:3054] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:57:55.025254 17984 raft_consensus.cc:513] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a675eb3dba403a8cd3f725f30a4c37" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35493 } }
I20250109 20:57:55.026062 17984 leader_election.cc:304] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4a675eb3dba403a8cd3f725f30a4c37; no voters:
I20250109 20:57:55.026750 17984 leader_election.cc:290] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:57:55.027173 17994 raft_consensus.cc:2798] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:57:55.032204 17994 raft_consensus.cc:695] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [term 1 LEADER]: Becoming Leader. State: Replica: e4a675eb3dba403a8cd3f725f30a4c37, State: Running, Role: LEADER
I20250109 20:57:55.032934 17994 consensus_queue.cc:237] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a675eb3dba403a8cd3f725f30a4c37" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35493 } }
I20250109 20:57:55.038267 17984 ts_tablet_manager.cc:1428] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37: Time spent starting tablet: real 0.047s user 0.013s sys 0.003s
I20250109 20:57:55.061332 17628 catalog_manager.cc:5526] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4a675eb3dba403a8cd3f725f30a4c37 (127.17.14.129). New cstate: current_term: 1 leader_uuid: "e4a675eb3dba403a8cd3f725f30a4c37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a675eb3dba403a8cd3f725f30a4c37" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35493 } health_report { overall_health: HEALTHY } } }
I20250109 20:57:55.107599 17466 tablet_server.cc:178] TabletServer@127.17.14.129:0 shutting down...
I20250109 20:57:55.165782 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:57:55.167248 17466 tablet_replica.cc:331] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37: stopping tablet replica
I20250109 20:57:55.168032 17466 raft_consensus.cc:2238] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:57:55.168610 17466 raft_consensus.cc:2267] T c7e0c6e2f8a54737887da84813c6faee P e4a675eb3dba403a8cd3f725f30a4c37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:57:55.172286 17466 tablet_replica.cc:331] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37: stopping tablet replica
I20250109 20:57:55.172888 17466 raft_consensus.cc:2238] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:57:55.173370 17466 raft_consensus.cc:2267] T f2529fbe4ada45e08d322e0e1dfe2426 P e4a675eb3dba403a8cd3f725f30a4c37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:57:55.405197 17466 tablet_server.cc:195] TabletServer@127.17.14.129:0 shutdown complete.
I20250109 20:57:55.420964 17466 tablet_server.cc:178] TabletServer@127.17.14.130:0 shutting down...
I20250109 20:57:55.483521 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:57:55.484679 17466 tablet_replica.cc:331] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a: stopping tablet replica
I20250109 20:57:55.485531 17466 raft_consensus.cc:2238] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:57:55.486138 17466 raft_consensus.cc:2267] T 25e059e377a646ad934df7702012572b P 4a59e879d70845a79c27bcad5e0abf6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:57:56.350188 17701 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:57:56.350656 17701 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250109 20:57:56.523011 17466 thread.cc:535] Waited for 1000ms trying to join with diag-logger (tid 17868)
W20250109 20:57:57.523509 17466 thread.cc:535] Waited for 2000ms trying to join with diag-logger (tid 17868)
I20250109 20:57:58.351667 17701 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:57:58.351984 17701 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:57:58.492023 17466 tablet_server.cc:195] TabletServer@127.17.14.130:0 shutdown complete.
I20250109 20:57:58.499485 17466 tablet_server.cc:178] TabletServer@127.17.14.131:0 shutting down...
I20250109 20:57:58.516444 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:57:58.516903 17466 tablet_replica.cc:331] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76: stopping tablet replica
I20250109 20:57:58.517315 17466 raft_consensus.cc:2238] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:57:58.517683 17466 raft_consensus.cc:2267] T da37c46e01cb413da83de231936bc15b P 1fa1800445b04f909f6aaf0141308e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:57:58.534407 17466 tablet_server.cc:195] TabletServer@127.17.14.131:0 shutdown complete.
I20250109 20:57:58.541378 17466 master.cc:537] Master@127.17.14.190:34671 shutting down...
I20250109 20:57:58.587718 17707 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20250109 20:57:58.626142 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:57:58.626667 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:57:58.626971 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2552a73bef9e45a0bc674029ae573cdc: stopping tablet replica
I20250109 20:57:58.646154 17466 master.cc:559] Master@127.17.14.190:34671 shutdown complete.
I20250109 20:57:58.654852 17466 master.cc:537] Master@127.17.14.189:34769 shutting down...
I20250109 20:57:58.668298 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:57:58.668702 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:57:58.668927 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P 5370928dcee047f4be450ed478f6a26f: stopping tablet replica
I20250109 20:57:58.728169 17466 master.cc:559] Master@127.17.14.189:34769 shutdown complete.
I20250109 20:57:58.737172 17466 master.cc:537] Master@127.17.14.188:45963 shutting down...
I20250109 20:57:58.750967 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:57:58.751749 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:57:58.752024 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P 4bb8f00c576b46abaddaf5ff786da3e2: stopping tablet replica
I20250109 20:57:58.769135 17466 master.cc:559] Master@127.17.14.188:45963 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (8191 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20250109 20:57:58.794560 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:45483
I20250109 20:57:58.795490 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:57:58.800360 18009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:58.801149 18010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:58.801568 18012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:58.802587 17466 server_base.cc:1034] running on GCE node
I20250109 20:57:58.803221 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:57:58.803402 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:57:58.803520 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456278803510 us; error 0 us; skew 500 ppm
I20250109 20:57:58.803918 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:57:58.811085 17466 webserver.cc:458] Webserver started at http://127.17.14.190:43869/ using document root <none> and password file <none>
I20250109 20:57:58.811530 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:57:58.811700 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:57:58.811937 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:57:58.812916 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "6d150a348e194a0abce748c10c8fb2b4"
format_stamp: "Formatted at 2025-01-09 20:57:58 on dist-test-slave-j54z"
I20250109 20:57:58.816938 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20250109 20:57:58.819942 18017 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:58.820606 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 20:57:58.820853 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "6d150a348e194a0abce748c10c8fb2b4"
format_stamp: "Formatted at 2025-01-09 20:57:58 on dist-test-slave-j54z"
I20250109 20:57:58.821123 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:57:58.836393 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:57:58.837297 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:57:58.869238 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:45483
I20250109 20:57:58.869326 18068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:45483 every 8 connection(s)
I20250109 20:57:58.872686 18069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:57:58.881268 18069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4: Bootstrap starting.
I20250109 20:57:58.885075 18069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4: Neither blocks nor log segments found. Creating new log.
I20250109 20:57:58.888631 18069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4: No bootstrap required, opened a new log
I20250109 20:57:58.890363 18069 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d150a348e194a0abce748c10c8fb2b4" member_type: VOTER }
I20250109 20:57:58.890715 18069 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:57:58.890937 18069 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d150a348e194a0abce748c10c8fb2b4, State: Initialized, Role: FOLLOWER
I20250109 20:57:58.891495 18069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d150a348e194a0abce748c10c8fb2b4" member_type: VOTER }
I20250109 20:57:58.891927 18069 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:57:58.892131 18069 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:57:58.892379 18069 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:57:58.896040 18069 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d150a348e194a0abce748c10c8fb2b4" member_type: VOTER }
I20250109 20:57:58.896510 18069 leader_election.cc:304] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d150a348e194a0abce748c10c8fb2b4; no voters:
I20250109 20:57:58.897653 18069 leader_election.cc:290] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:57:58.897893 18072 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:57:58.899267 18072 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [term 1 LEADER]: Becoming Leader. State: Replica: 6d150a348e194a0abce748c10c8fb2b4, State: Running, Role: LEADER
I20250109 20:57:58.899873 18072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d150a348e194a0abce748c10c8fb2b4" member_type: VOTER }
I20250109 20:57:58.900494 18069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:57:58.904100 18074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d150a348e194a0abce748c10c8fb2b4. Latest consensus state: current_term: 1 leader_uuid: "6d150a348e194a0abce748c10c8fb2b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d150a348e194a0abce748c10c8fb2b4" member_type: VOTER } }
I20250109 20:57:58.904836 18074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [sys.catalog]: This master's current role is: LEADER
I20250109 20:57:58.905380 18073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d150a348e194a0abce748c10c8fb2b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d150a348e194a0abce748c10c8fb2b4" member_type: VOTER } }
I20250109 20:57:58.906018 18073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [sys.catalog]: This master's current role is: LEADER
I20250109 20:57:58.907217 18079 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:57:58.912391 18079 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:57:58.916934 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 20:57:58.921803 18079 catalog_manager.cc:1348] Generated new cluster ID: 52adc1f442fa4ed28478fe33c7b876e5
I20250109 20:57:58.922086 18079 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:57:58.937151 18079 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 20:57:58.938302 18079 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:57:58.954720 18079 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4: Generated new TSK 0
I20250109 20:57:58.955278 18079 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:57:58.983454 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:57:58.989104 18090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:58.990134 18091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:58.991219 18093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:58.992095 17466 server_base.cc:1034] running on GCE node
I20250109 20:57:58.992810 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:57:58.992981 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:57:58.993106 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456278993096 us; error 0 us; skew 500 ppm
I20250109 20:57:58.993597 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:57:58.995792 17466 webserver.cc:458] Webserver started at http://127.17.14.129:36433/ using document root <none> and password file <none>
I20250109 20:57:58.996199 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:57:58.996354 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:57:58.996572 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:57:58.997609 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-0-root/instance:
uuid: "c84ca066a3a647558af06e1ddf72cfcb"
format_stamp: "Formatted at 2025-01-09 20:57:58 on dist-test-slave-j54z"
I20250109 20:57:59.001646 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250109 20:57:59.004644 18098 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:59.005342 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:57:59.005654 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "c84ca066a3a647558af06e1ddf72cfcb"
format_stamp: "Formatted at 2025-01-09 20:57:58 on dist-test-slave-j54z"
I20250109 20:57:59.005913 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:57:59.022521 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:57:59.023562 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:57:59.024889 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:57:59.027067 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:57:59.027257 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:59.027491 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:57:59.027639 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:59.063689 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:44553
I20250109 20:57:59.063781 18160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:44553 every 8 connection(s)
I20250109 20:57:59.068038 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:57:59.074994 18165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:59.075968 18166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:59.077554 18168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:59.078770 17466 server_base.cc:1034] running on GCE node
I20250109 20:57:59.079716 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250109 20:57:59.079831 18161 heartbeater.cc:346] Connected to a master server at 127.17.14.190:45483
W20250109 20:57:59.079975 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:57:59.080184 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456279080169 us; error 0 us; skew 500 ppm
I20250109 20:57:59.080200 18161 heartbeater.cc:463] Registering TS with master...
I20250109 20:57:59.080758 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:57:59.080991 18161 heartbeater.cc:510] Master 127.17.14.190:45483 requested a full tablet report, sending...
I20250109 20:57:59.083235 18034 ts_manager.cc:194] Registered new tserver with Master: c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553)
I20250109 20:57:59.083351 17466 webserver.cc:458] Webserver started at http://127.17.14.130:38183/ using document root <none> and password file <none>
I20250109 20:57:59.083869 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:57:59.084053 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:57:59.084286 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:57:59.084926 18034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37978
I20250109 20:57:59.085387 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-1-root/instance:
uuid: "45c1b06aef574962914c99b98dcbd5fe"
format_stamp: "Formatted at 2025-01-09 20:57:59 on dist-test-slave-j54z"
I20250109 20:57:59.089540 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250109 20:57:59.093290 18173 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:59.094053 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:57:59.094337 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "45c1b06aef574962914c99b98dcbd5fe"
format_stamp: "Formatted at 2025-01-09 20:57:59 on dist-test-slave-j54z"
I20250109 20:57:59.094614 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:57:59.105522 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:57:59.106417 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:57:59.107784 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:57:59.109934 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:57:59.110112 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:59.110343 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:57:59.110493 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:59.145566 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:36803
I20250109 20:57:59.145692 18235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:36803 every 8 connection(s)
I20250109 20:57:59.149789 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:57:59.156332 18239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:57:59.158526 18240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:59.160416 17466 server_base.cc:1034] running on GCE node
I20250109 20:57:59.161177 18236 heartbeater.cc:346] Connected to a master server at 127.17.14.190:45483
I20250109 20:57:59.161513 18236 heartbeater.cc:463] Registering TS with master...
W20250109 20:57:59.161757 18242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:57:59.162187 18236 heartbeater.cc:510] Master 127.17.14.190:45483 requested a full tablet report, sending...
I20250109 20:57:59.162506 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:57:59.162727 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:57:59.162900 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456279162883 us; error 0 us; skew 500 ppm
I20250109 20:57:59.163614 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:57:59.164155 18034 ts_manager.cc:194] Registered new tserver with Master: 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:57:59.165779 18034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37994
I20250109 20:57:59.166391 17466 webserver.cc:458] Webserver started at http://127.17.14.131:33863/ using document root <none> and password file <none>
I20250109 20:57:59.166935 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:57:59.167120 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:57:59.167366 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:57:59.168318 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-2-root/instance:
uuid: "c33d4d09803d48f99f5f4bf8ac33d402"
format_stamp: "Formatted at 2025-01-09 20:57:59 on dist-test-slave-j54z"
I20250109 20:57:59.172381 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 20:57:59.175279 18247 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:59.175928 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 20:57:59.176172 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "c33d4d09803d48f99f5f4bf8ac33d402"
format_stamp: "Formatted at 2025-01-09 20:57:59 on dist-test-slave-j54z"
I20250109 20:57:59.176445 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:57:59.204362 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:57:59.205415 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:57:59.206763 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:57:59.208990 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:57:59.209173 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:57:59.209430 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:57:59.209667 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250109 20:57:59.244148 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:33533
I20250109 20:57:59.244235 18309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:33533 every 8 connection(s)
I20250109 20:57:59.256116 18310 heartbeater.cc:346] Connected to a master server at 127.17.14.190:45483
I20250109 20:57:59.256489 18310 heartbeater.cc:463] Registering TS with master...
I20250109 20:57:59.257149 18310 heartbeater.cc:510] Master 127.17.14.190:45483 requested a full tablet report, sending...
I20250109 20:57:59.258846 18034 ts_manager.cc:194] Registered new tserver with Master: c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:57:59.258931 17466 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012019809s
I20250109 20:57:59.260051 18034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37996
I20250109 20:58:00.087172 18161 heartbeater.cc:502] Master 127.17.14.190:45483 was elected leader, sending a full tablet report...
I20250109 20:58:00.168128 18236 heartbeater.cc:502] Master 127.17.14.190:45483 was elected leader, sending a full tablet report...
I20250109 20:58:00.262207 18310 heartbeater.cc:502] Master 127.17.14.190:45483 was elected leader, sending a full tablet report...
I20250109 20:58:00.290650 17466 test_util.cc:274] Using random seed: 1002517795
I20250109 20:58:00.311187 18034 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38010:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:58:00.313611 18034 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:58:00.392007 18126 tablet_service.cc:1467] Processing CreateTablet for tablet bec5186b23f94af985e2ede96c30b0dc (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250109 20:58:00.393335 18126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bec5186b23f94af985e2ede96c30b0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.394155 18125 tablet_service.cc:1467] Processing CreateTablet for tablet 1e6f9e2410d440079d9ff8e622c7369a (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250109 20:58:00.394520 18120 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd2b37a3e154ffea27c42bd78441e4a (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250109 20:58:00.395416 18125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e6f9e2410d440079d9ff8e622c7369a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.396534 18121 tablet_service.cc:1467] Processing CreateTablet for tablet 762caa2bc7564c9d88448466f319a2bf (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250109 20:58:00.397706 18119 tablet_service.cc:1467] Processing CreateTablet for tablet 7500274b7719441596814acc45fa3028 (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250109 20:58:00.398713 18122 tablet_service.cc:1467] Processing CreateTablet for tablet a9942465bba148559c8310fecb210d09 (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250109 20:58:00.399941 18122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9942465bba148559c8310fecb210d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.401867 18119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7500274b7719441596814acc45fa3028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.402179 18123 tablet_service.cc:1467] Processing CreateTablet for tablet 19f8261bc37f4e8ea89f4c02dc3c4478 (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250109 20:58:00.403214 18121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 762caa2bc7564c9d88448466f319a2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.404161 18124 tablet_service.cc:1467] Processing CreateTablet for tablet 40bb82b56471437eba54b55ebb6c20ce (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250109 20:58:00.405373 18124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40bb82b56471437eba54b55ebb6c20ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.408916 18120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bd2b37a3e154ffea27c42bd78441e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.410087 18123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f8261bc37f4e8ea89f4c02dc3c4478. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.413333 18201 tablet_service.cc:1467] Processing CreateTablet for tablet bec5186b23f94af985e2ede96c30b0dc (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250109 20:58:00.414616 18201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bec5186b23f94af985e2ede96c30b0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.415937 18274 tablet_service.cc:1467] Processing CreateTablet for tablet 1e6f9e2410d440079d9ff8e622c7369a (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250109 20:58:00.416465 18271 tablet_service.cc:1467] Processing CreateTablet for tablet a9942465bba148559c8310fecb210d09 (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250109 20:58:00.417223 18274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e6f9e2410d440079d9ff8e622c7369a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.418274 18194 tablet_service.cc:1467] Processing CreateTablet for tablet 7500274b7719441596814acc45fa3028 (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250109 20:58:00.418682 18195 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd2b37a3e154ffea27c42bd78441e4a (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250109 20:58:00.419782 18194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7500274b7719441596814acc45fa3028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.420953 18196 tablet_service.cc:1467] Processing CreateTablet for tablet 762caa2bc7564c9d88448466f319a2bf (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250109 20:58:00.422179 18196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 762caa2bc7564c9d88448466f319a2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.418694 18273 tablet_service.cc:1467] Processing CreateTablet for tablet 40bb82b56471437eba54b55ebb6c20ce (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250109 20:58:00.426788 18273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40bb82b56471437eba54b55ebb6c20ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.426533 18268 tablet_service.cc:1467] Processing CreateTablet for tablet 7500274b7719441596814acc45fa3028 (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250109 20:58:00.427681 18268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7500274b7719441596814acc45fa3028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.430454 18197 tablet_service.cc:1467] Processing CreateTablet for tablet a9942465bba148559c8310fecb210d09 (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250109 20:58:00.431638 18197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9942465bba148559c8310fecb210d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.414297 18275 tablet_service.cc:1467] Processing CreateTablet for tablet bec5186b23f94af985e2ede96c30b0dc (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250109 20:58:00.434554 18269 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd2b37a3e154ffea27c42bd78441e4a (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250109 20:58:00.435461 18275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bec5186b23f94af985e2ede96c30b0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.436803 18270 tablet_service.cc:1467] Processing CreateTablet for tablet 762caa2bc7564c9d88448466f319a2bf (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250109 20:58:00.438016 18270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 762caa2bc7564c9d88448466f319a2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.446957 18272 tablet_service.cc:1467] Processing CreateTablet for tablet 19f8261bc37f4e8ea89f4c02dc3c4478 (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250109 20:58:00.449683 18200 tablet_service.cc:1467] Processing CreateTablet for tablet 1e6f9e2410d440079d9ff8e622c7369a (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250109 20:58:00.454452 18198 tablet_service.cc:1467] Processing CreateTablet for tablet 19f8261bc37f4e8ea89f4c02dc3c4478 (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250109 20:58:00.462234 18199 tablet_service.cc:1467] Processing CreateTablet for tablet 40bb82b56471437eba54b55ebb6c20ce (DEFAULT_TABLE table=test-workload [id=7553662f6e064f84ac8500a0afb6e669]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250109 20:58:00.467602 18269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bd2b37a3e154ffea27c42bd78441e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.467650 18195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bd2b37a3e154ffea27c42bd78441e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.475297 18200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e6f9e2410d440079d9ff8e622c7369a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.476218 18199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40bb82b56471437eba54b55ebb6c20ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.477885 18198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f8261bc37f4e8ea89f4c02dc3c4478. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.479568 18271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9942465bba148559c8310fecb210d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.483275 18330 tablet_bootstrap.cc:492] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb: Bootstrap starting.
I20250109 20:58:00.483366 18272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f8261bc37f4e8ea89f4c02dc3c4478. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:00.532543 18331 tablet_bootstrap.cc:492] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402: Bootstrap starting.
I20250109 20:58:00.538237 18331 tablet_bootstrap.cc:654] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.543043 18330 tablet_bootstrap.cc:654] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.551155 18332 tablet_bootstrap.cc:492] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe: Bootstrap starting.
I20250109 20:58:00.553117 18330 tablet_bootstrap.cc:492] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb: No bootstrap required, opened a new log
I20250109 20:58:00.553596 18330 ts_tablet_manager.cc:1397] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.010s
I20250109 20:58:00.555702 18331 tablet_bootstrap.cc:492] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402: No bootstrap required, opened a new log
I20250109 20:58:00.556099 18331 ts_tablet_manager.cc:1397] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.006s
I20250109 20:58:00.556277 18330 raft_consensus.cc:357] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:00.557099 18330 raft_consensus.cc:383] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.557399 18330 raft_consensus.cc:738] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ca066a3a647558af06e1ddf72cfcb, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.558187 18331 raft_consensus.cc:357] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.558106 18330 consensus_queue.cc:260] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:00.558804 18331 raft_consensus.cc:383] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.559113 18331 raft_consensus.cc:738] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.559168 18332 tablet_bootstrap.cc:654] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.559720 18331 consensus_queue.cc:260] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.562374 18331 ts_tablet_manager.cc:1428] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250109 20:58:00.563244 18331 tablet_bootstrap.cc:492] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402: Bootstrap starting.
I20250109 20:58:00.568069 18330 ts_tablet_manager.cc:1428] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20250109 20:58:00.568547 18331 tablet_bootstrap.cc:654] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.569712 18330 tablet_bootstrap.cc:492] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb: Bootstrap starting.
I20250109 20:58:00.575313 18330 tablet_bootstrap.cc:654] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.578722 18331 tablet_bootstrap.cc:492] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402: No bootstrap required, opened a new log
I20250109 20:58:00.579162 18331 ts_tablet_manager.cc:1397] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20250109 20:58:00.579334 18332 tablet_bootstrap.cc:492] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe: No bootstrap required, opened a new log
I20250109 20:58:00.579689 18332 ts_tablet_manager.cc:1397] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.000s
I20250109 20:58:00.580947 18331 raft_consensus.cc:357] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.581516 18331 raft_consensus.cc:383] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.581354 18332 raft_consensus.cc:357] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.581825 18331 raft_consensus.cc:738] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.582038 18332 raft_consensus.cc:383] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.582350 18332 raft_consensus.cc:738] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c1b06aef574962914c99b98dcbd5fe, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.582466 18331 consensus_queue.cc:260] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.582857 18332 consensus_queue.cc:260] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.584584 18331 ts_tablet_manager.cc:1428] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20250109 20:58:00.585443 18331 tablet_bootstrap.cc:492] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402: Bootstrap starting.
I20250109 20:58:00.585475 18332 ts_tablet_manager.cc:1428] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250109 20:58:00.586531 18332 tablet_bootstrap.cc:492] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe: Bootstrap starting.
I20250109 20:58:00.590135 18330 tablet_bootstrap.cc:492] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb: No bootstrap required, opened a new log
I20250109 20:58:00.590612 18330 ts_tablet_manager.cc:1397] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20250109 20:58:00.591307 18331 tablet_bootstrap.cc:654] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.591745 18332 tablet_bootstrap.cc:654] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.593026 18330 raft_consensus.cc:357] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.593824 18330 raft_consensus.cc:383] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.594151 18330 raft_consensus.cc:738] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ca066a3a647558af06e1ddf72cfcb, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.595006 18330 consensus_queue.cc:260] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.602413 18330 ts_tablet_manager.cc:1428] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250109 20:58:00.603675 18330 tablet_bootstrap.cc:492] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb: Bootstrap starting.
I20250109 20:58:00.610944 18330 tablet_bootstrap.cc:654] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.611784 18335 raft_consensus.cc:491] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:00.612361 18335 raft_consensus.cc:513] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:00.615429 18335 leader_election.cc:290] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:00.625365 18336 raft_consensus.cc:491] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:00.626036 18336 raft_consensus.cc:513] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.631354 18336 leader_election.cc:290] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:00.640482 18332 tablet_bootstrap.cc:492] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe: No bootstrap required, opened a new log
I20250109 20:58:00.641147 18332 ts_tablet_manager.cc:1397] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe: Time spent bootstrapping tablet: real 0.055s user 0.016s sys 0.029s
I20250109 20:58:00.644565 18332 raft_consensus.cc:357] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.645563 18332 raft_consensus.cc:383] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.645965 18332 raft_consensus.cc:738] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c1b06aef574962914c99b98dcbd5fe, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.646921 18332 consensus_queue.cc:260] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.648324 18331 tablet_bootstrap.cc:492] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402: No bootstrap required, opened a new log
I20250109 20:58:00.648941 18331 ts_tablet_manager.cc:1397] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402: Time spent bootstrapping tablet: real 0.064s user 0.011s sys 0.017s
I20250109 20:58:00.650146 18332 ts_tablet_manager.cc:1428] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250109 20:58:00.651067 18332 tablet_bootstrap.cc:492] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe: Bootstrap starting.
I20250109 20:58:00.658564 18332 tablet_bootstrap.cc:654] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.669588 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6f9e2410d440079d9ff8e622c7369a" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb" is_pre_election: true
W20250109 20:58:00.671556 18248 leader_election.cc:343] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:58:00.672407 18332 tablet_bootstrap.cc:492] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe: No bootstrap required, opened a new log
I20250109 20:58:00.673004 18332 ts_tablet_manager.cc:1397] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.000s
I20250109 20:58:00.663261 18331 raft_consensus.cc:357] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.676007 18331 raft_consensus.cc:383] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.675827 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec5186b23f94af985e2ede96c30b0dc" candidate_uuid: "c84ca066a3a647558af06e1ddf72cfcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" is_pre_election: true
I20250109 20:58:00.676314 18331 raft_consensus.cc:738] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Initialized, Role: FOLLOWER
W20250109 20:58:00.677435 18101 leader_election.cc:343] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:58:00.680444 18331 consensus_queue.cc:260] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.681703 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6f9e2410d440079d9ff8e622c7369a" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe" is_pre_election: true
I20250109 20:58:00.682085 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec5186b23f94af985e2ede96c30b0dc" candidate_uuid: "c84ca066a3a647558af06e1ddf72cfcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe" is_pre_election: true
I20250109 20:58:00.683367 18331 ts_tablet_manager.cc:1428] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402: Time spent starting tablet: real 0.034s user 0.009s sys 0.000s
W20250109 20:58:00.683728 18250 leader_election.cc:343] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:58:00.684316 18331 tablet_bootstrap.cc:492] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402: Bootstrap starting.
I20250109 20:58:00.684168 18250 leader_election.cc:304] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c33d4d09803d48f99f5f4bf8ac33d402; no voters: 45c1b06aef574962914c99b98dcbd5fe, c84ca066a3a647558af06e1ddf72cfcb
I20250109 20:58:00.684631 18330 tablet_bootstrap.cc:492] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb: No bootstrap required, opened a new log
I20250109 20:58:00.685078 18330 ts_tablet_manager.cc:1397] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb: Time spent bootstrapping tablet: real 0.082s user 0.029s sys 0.047s
I20250109 20:58:00.685156 18336 raft_consensus.cc:2743] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:58:00.685416 18101 leader_election.cc:343] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:58:00.681815 18332 raft_consensus.cc:357] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.686445 18332 raft_consensus.cc:383] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.686900 18332 raft_consensus.cc:738] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c1b06aef574962914c99b98dcbd5fe, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.688030 18330 raft_consensus.cc:357] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.687834 18332 consensus_queue.cc:260] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.688691 18330 raft_consensus.cc:383] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.689021 18330 raft_consensus.cc:738] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ca066a3a647558af06e1ddf72cfcb, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.689687 18330 consensus_queue.cc:260] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.690569 18336 raft_consensus.cc:491] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:00.690799 18331 tablet_bootstrap.cc:654] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.691135 18336 raft_consensus.cc:513] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.691905 18330 ts_tablet_manager.cc:1428] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250109 20:58:00.692823 18330 tablet_bootstrap.cc:492] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb: Bootstrap starting.
I20250109 20:58:00.693218 18336 leader_election.cc:290] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:00.694545 18332 ts_tablet_manager.cc:1428] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20250109 20:58:00.695508 18332 tablet_bootstrap.cc:492] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe: Bootstrap starting.
I20250109 20:58:00.698135 18330 tablet_bootstrap.cc:654] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.685912 18101 leader_election.cc:304] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c84ca066a3a647558af06e1ddf72cfcb; no voters: 45c1b06aef574962914c99b98dcbd5fe, c33d4d09803d48f99f5f4bf8ac33d402
I20250109 20:58:00.725976 18335 raft_consensus.cc:2743] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:58:00.726030 18331 tablet_bootstrap.cc:492] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402: No bootstrap required, opened a new log
I20250109 20:58:00.728595 18331 ts_tablet_manager.cc:1397] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.006s
I20250109 20:58:00.733500 18332 tablet_bootstrap.cc:654] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.734400 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762caa2bc7564c9d88448466f319a2bf" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb" is_pre_election: true
I20250109 20:58:00.735672 18136 raft_consensus.cc:2463] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 0.
I20250109 20:58:00.733582 18340 raft_consensus.cc:491] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:00.737793 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762caa2bc7564c9d88448466f319a2bf" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe" is_pre_election: true
I20250109 20:58:00.738281 18340 raft_consensus.cc:513] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.739364 18336 raft_consensus.cc:491] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:00.740078 18336 raft_consensus.cc:513] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.741461 18331 raft_consensus.cc:357] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.742370 18336 leader_election.cc:290] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:00.742558 18331 raft_consensus.cc:383] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.743062 18331 raft_consensus.cc:738] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.744037 18331 consensus_queue.cc:260] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.748100 18331 ts_tablet_manager.cc:1428] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402: Time spent starting tablet: real 0.019s user 0.004s sys 0.007s
I20250109 20:58:00.750703 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bb82b56471437eba54b55ebb6c20ce" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe" is_pre_election: true
I20250109 20:58:00.751500 18211 raft_consensus.cc:2463] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 0.
I20250109 20:58:00.752478 18210 raft_consensus.cc:2463] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 0.
I20250109 20:58:00.756098 18340 leader_election.cc:290] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:00.758054 18248 leader_election.cc:304] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c33d4d09803d48f99f5f4bf8ac33d402, c84ca066a3a647558af06e1ddf72cfcb; no voters:
I20250109 20:58:00.760344 18250 leader_election.cc:304] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c33d4d09803d48f99f5f4bf8ac33d402; no voters:
I20250109 20:58:00.763286 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bb82b56471437eba54b55ebb6c20ce" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb" is_pre_election: true
I20250109 20:58:00.766045 18336 raft_consensus.cc:2798] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:58:00.766459 18336 raft_consensus.cc:491] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:58:00.766822 18336 raft_consensus.cc:3054] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:00.782094 18331 tablet_bootstrap.cc:492] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402: Bootstrap starting.
W20250109 20:58:00.785409 18248 leader_election.cc:343] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:58:00.791232 18331 tablet_bootstrap.cc:654] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.795555 18336 raft_consensus.cc:513] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.798674 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762caa2bc7564c9d88448466f319a2bf" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
I20250109 20:58:00.799235 18136 raft_consensus.cc:3054] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:00.801435 18336 leader_election.cc:290] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Requested vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:00.801769 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762caa2bc7564c9d88448466f319a2bf" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe"
I20250109 20:58:00.802464 18210 raft_consensus.cc:3054] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:00.807638 18135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762caa2bc7564c9d88448466f319a2bf" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb" is_pre_election: true
I20250109 20:58:00.810357 18136 raft_consensus.cc:2463] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 1.
I20250109 20:58:00.811519 18248 leader_election.cc:304] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c33d4d09803d48f99f5f4bf8ac33d402, c84ca066a3a647558af06e1ddf72cfcb; no voters:
I20250109 20:58:00.811964 18332 tablet_bootstrap.cc:492] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe: No bootstrap required, opened a new log
I20250109 20:58:00.812222 18330 tablet_bootstrap.cc:492] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb: No bootstrap required, opened a new log
I20250109 20:58:00.812762 18332 ts_tablet_manager.cc:1397] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe: Time spent bootstrapping tablet: real 0.117s user 0.026s sys 0.021s
I20250109 20:58:00.813043 18330 ts_tablet_manager.cc:1397] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb: Time spent bootstrapping tablet: real 0.120s user 0.023s sys 0.039s
I20250109 20:58:00.813417 18336 raft_consensus.cc:2798] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:00.814080 18336 raft_consensus.cc:695] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Becoming Leader. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Running, Role: LEADER
I20250109 20:58:00.813634 18210 raft_consensus.cc:2463] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 1.
I20250109 20:58:00.815793 18336 consensus_queue.cc:237] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.816071 18332 raft_consensus.cc:357] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:00.817266 18332 raft_consensus.cc:383] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.817060 18330 raft_consensus.cc:357] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.817653 18332 raft_consensus.cc:738] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c1b06aef574962914c99b98dcbd5fe, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.811977 18356 raft_consensus.cc:2798] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:58:00.818518 18356 raft_consensus.cc:491] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:58:00.818859 18356 raft_consensus.cc:3054] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:00.818737 18332 consensus_queue.cc:260] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:00.817811 18330 raft_consensus.cc:383] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.825672 18330 raft_consensus.cc:738] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ca066a3a647558af06e1ddf72cfcb, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.826596 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762caa2bc7564c9d88448466f319a2bf" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" is_pre_election: true
I20250109 20:58:00.826797 18330 consensus_queue.cc:260] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.828622 18176 leader_election.cc:304] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe; no voters: c33d4d09803d48f99f5f4bf8ac33d402, c84ca066a3a647558af06e1ddf72cfcb
I20250109 20:58:00.829603 18340 raft_consensus.cc:2743] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:58:00.831832 18332 ts_tablet_manager.cc:1428] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250109 20:58:00.836050 18331 tablet_bootstrap.cc:492] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402: No bootstrap required, opened a new log
I20250109 20:58:00.836355 18330 ts_tablet_manager.cc:1428] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb: Time spent starting tablet: real 0.023s user 0.003s sys 0.003s
I20250109 20:58:00.836572 18331 ts_tablet_manager.cc:1397] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.005s
I20250109 20:58:00.835558 18032 catalog_manager.cc:5526] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 reported cstate change: term changed from 0 to 1, leader changed from <none> to c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131). New cstate: current_term: 1 leader_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } health_report { overall_health: HEALTHY } } }
I20250109 20:58:00.838178 18330 tablet_bootstrap.cc:492] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb: Bootstrap starting.
I20250109 20:58:00.839570 18331 raft_consensus.cc:357] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.840433 18331 raft_consensus.cc:383] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.840727 18331 raft_consensus.cc:738] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.841763 18331 consensus_queue.cc:260] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.844159 18331 ts_tablet_manager.cc:1428] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250109 20:58:00.845407 18331 tablet_bootstrap.cc:492] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402: Bootstrap starting.
I20250109 20:58:00.846072 18356 raft_consensus.cc:513] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.846783 18330 tablet_bootstrap.cc:654] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.849128 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bb82b56471437eba54b55ebb6c20ce" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
W20250109 20:58:00.850713 18248 leader_election.cc:343] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:58:00.852106 18331 tablet_bootstrap.cc:654] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.853247 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bb82b56471437eba54b55ebb6c20ce" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe"
I20250109 20:58:00.854058 18210 raft_consensus.cc:3054] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:00.855839 18332 tablet_bootstrap.cc:492] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe: Bootstrap starting.
I20250109 20:58:00.855897 18356 leader_election.cc:290] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Requested vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:00.864820 18210 raft_consensus.cc:2463] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 1.
I20250109 20:58:00.866320 18250 leader_election.cc:304] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c33d4d09803d48f99f5f4bf8ac33d402; no voters: c84ca066a3a647558af06e1ddf72cfcb
I20250109 20:58:00.867802 18356 raft_consensus.cc:2798] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:00.868463 18356 raft_consensus.cc:695] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Becoming Leader. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Running, Role: LEADER
I20250109 20:58:00.869747 18356 consensus_queue.cc:237] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.877137 18332 tablet_bootstrap.cc:654] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.878867 18330 tablet_bootstrap.cc:492] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb: No bootstrap required, opened a new log
I20250109 20:58:00.879144 18331 tablet_bootstrap.cc:492] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402: No bootstrap required, opened a new log
I20250109 20:58:00.879283 18330 ts_tablet_manager.cc:1397] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb: Time spent bootstrapping tablet: real 0.041s user 0.018s sys 0.020s
I20250109 20:58:00.879622 18331 ts_tablet_manager.cc:1397] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402: Time spent bootstrapping tablet: real 0.035s user 0.021s sys 0.008s
I20250109 20:58:00.882042 18330 raft_consensus.cc:357] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.882928 18330 raft_consensus.cc:383] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.883529 18330 raft_consensus.cc:738] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ca066a3a647558af06e1ddf72cfcb, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.882895 18331 raft_consensus.cc:357] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.884042 18331 raft_consensus.cc:383] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.882884 18332 tablet_bootstrap.cc:492] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe: No bootstrap required, opened a new log
I20250109 20:58:00.884358 18331 raft_consensus.cc:738] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.884975 18332 ts_tablet_manager.cc:1397] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.004s
I20250109 20:58:00.884424 18330 consensus_queue.cc:260] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.885277 18331 consensus_queue.cc:260] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.887892 18331 ts_tablet_manager.cc:1428] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250109 20:58:00.887991 18332 raft_consensus.cc:357] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.888989 18332 raft_consensus.cc:383] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:00.889113 18331 tablet_bootstrap.cc:492] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402: Bootstrap starting.
I20250109 20:58:00.888546 18034 catalog_manager.cc:5526] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 reported cstate change: term changed from 0 to 1, leader changed from <none> to c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131). New cstate: current_term: 1 leader_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } health_report { overall_health: HEALTHY } } }
I20250109 20:58:00.889415 18332 raft_consensus.cc:738] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c1b06aef574962914c99b98dcbd5fe, State: Initialized, Role: FOLLOWER
I20250109 20:58:00.890300 18332 consensus_queue.cc:260] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.890972 18330 ts_tablet_manager.cc:1428] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20250109 20:58:00.892164 18330 tablet_bootstrap.cc:492] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb: Bootstrap starting.
I20250109 20:58:00.892885 18332 ts_tablet_manager.cc:1428] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250109 20:58:00.894078 18332 tablet_bootstrap.cc:492] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe: Bootstrap starting.
I20250109 20:58:00.896925 18331 tablet_bootstrap.cc:654] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.898620 18330 tablet_bootstrap.cc:654] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.903411 18332 tablet_bootstrap.cc:654] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:00.903925 18356 raft_consensus.cc:491] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:00.904425 18356 raft_consensus.cc:513] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.906520 18356 leader_election.cc:290] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:00.911387 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9942465bba148559c8310fecb210d09" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb" is_pre_election: true
I20250109 20:58:00.912581 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9942465bba148559c8310fecb210d09" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe" is_pre_election: true
I20250109 20:58:00.917461 18087 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250109 20:58:00.917860 18248 leader_election.cc:343] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:58:00.918725 18087 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250109 20:58:00.919370 18082 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250109 20:58:00.913206 18210 raft_consensus.cc:2463] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 0.
I20250109 20:58:00.920760 18250 leader_election.cc:304] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c33d4d09803d48f99f5f4bf8ac33d402; no voters: c84ca066a3a647558af06e1ddf72cfcb
I20250109 20:58:00.921845 18336 raft_consensus.cc:491] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:00.922389 18336 raft_consensus.cc:513] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.924862 18336 leader_election.cc:290] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:00.925460 18336 raft_consensus.cc:2798] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:58:00.933059 18336 raft_consensus.cc:491] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:58:00.933461 18336 raft_consensus.cc:3054] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:00.926921 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd2b37a3e154ffea27c42bd78441e4a" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb" is_pre_election: true
I20250109 20:58:00.938407 18136 raft_consensus.cc:2463] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 0.
I20250109 20:58:00.930904 18357 raft_consensus.cc:491] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:00.939802 18357 raft_consensus.cc:513] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.942466 18357 leader_election.cc:290] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:00.931859 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd2b37a3e154ffea27c42bd78441e4a" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe" is_pre_election: true
I20250109 20:58:00.946899 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7500274b7719441596814acc45fa3028" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe" is_pre_election: true
I20250109 20:58:00.948726 18248 leader_election.cc:304] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c33d4d09803d48f99f5f4bf8ac33d402, c84ca066a3a647558af06e1ddf72cfcb; no voters:
W20250109 20:58:00.949923 18250 leader_election.cc:343] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:58:00.950982 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7500274b7719441596814acc45fa3028" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb" is_pre_election: true
I20250109 20:58:00.951931 18136 raft_consensus.cc:2463] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 0.
I20250109 20:58:00.945097 18210 raft_consensus.cc:2463] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 0.
I20250109 20:58:00.954891 18248 leader_election.cc:304] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c33d4d09803d48f99f5f4bf8ac33d402, c84ca066a3a647558af06e1ddf72cfcb; no voters: 45c1b06aef574962914c99b98dcbd5fe
I20250109 20:58:00.956327 18356 raft_consensus.cc:2798] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:58:00.956807 18356 raft_consensus.cc:491] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:58:00.957218 18356 raft_consensus.cc:3054] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:00.962447 18357 raft_consensus.cc:2798] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:58:00.962867 18357 raft_consensus.cc:491] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:58:00.963197 18357 raft_consensus.cc:3054] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:00.979630 18356 raft_consensus.cc:513] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.982827 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd2b37a3e154ffea27c42bd78441e4a" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
I20250109 20:58:00.983588 18136 raft_consensus.cc:3054] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:00.992355 18336 raft_consensus.cc:513] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:00.996094 18135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9942465bba148559c8310fecb210d09" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
W20250109 20:58:00.997862 18248 leader_election.cc:343] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:58:00.976985 18357 raft_consensus.cc:513] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.004103 18135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7500274b7719441596814acc45fa3028" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
I20250109 20:58:01.004866 18135 raft_consensus.cc:3054] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.015388 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd2b37a3e154ffea27c42bd78441e4a" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe"
I20250109 20:58:01.016177 18210 raft_consensus.cc:3054] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.021575 18356 leader_election.cc:290] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Requested vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:01.026175 18135 raft_consensus.cc:2463] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 1.
I20250109 20:58:01.027729 18248 leader_election.cc:304] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c33d4d09803d48f99f5f4bf8ac33d402, c84ca066a3a647558af06e1ddf72cfcb; no voters:
I20250109 20:58:01.028831 18356 raft_consensus.cc:2798] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:01.028983 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7500274b7719441596814acc45fa3028" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe"
W20250109 20:58:01.030741 18250 leader_election.cc:343] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:58:01.025228 18210 raft_consensus.cc:2463] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 1.
I20250109 20:58:01.031324 18331 tablet_bootstrap.cc:492] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402: No bootstrap required, opened a new log
I20250109 20:58:01.031999 18331 ts_tablet_manager.cc:1397] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402: Time spent bootstrapping tablet: real 0.143s user 0.049s sys 0.080s
I20250109 20:58:01.032295 18250 leader_election.cc:304] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c33d4d09803d48f99f5f4bf8ac33d402; no voters:
I20250109 20:58:01.034885 18335 raft_consensus.cc:491] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:01.035468 18335 raft_consensus.cc:513] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.035706 18331 raft_consensus.cc:357] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:01.036665 18331 raft_consensus.cc:383] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:01.037050 18331 raft_consensus.cc:738] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Initialized, Role: FOLLOWER
I20250109 20:58:01.038034 18335 leader_election.cc:290] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803), c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:01.037928 18331 consensus_queue.cc:260] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:01.040802 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bb82b56471437eba54b55ebb6c20ce" candidate_uuid: "c84ca066a3a647558af06e1ddf72cfcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" is_pre_election: true
I20250109 20:58:01.042409 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bb82b56471437eba54b55ebb6c20ce" candidate_uuid: "c84ca066a3a647558af06e1ddf72cfcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe" is_pre_election: true
I20250109 20:58:01.043488 18211 raft_consensus.cc:2388] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c84ca066a3a647558af06e1ddf72cfcb in current term 1: Already voted for candidate c33d4d09803d48f99f5f4bf8ac33d402 in this term.
I20250109 20:58:01.045627 18101 leader_election.cc:304] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c84ca066a3a647558af06e1ddf72cfcb; no voters: 45c1b06aef574962914c99b98dcbd5fe, c33d4d09803d48f99f5f4bf8ac33d402
I20250109 20:58:01.046979 18335 raft_consensus.cc:3054] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.001098 18136 raft_consensus.cc:2463] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 1.
I20250109 20:58:01.057776 18357 leader_election.cc:290] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Requested vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:01.060310 18336 leader_election.cc:290] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Requested vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:01.061009 18336 raft_consensus.cc:2798] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:01.061532 18336 raft_consensus.cc:695] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Becoming Leader. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Running, Role: LEADER
I20250109 20:58:01.062484 18336 consensus_queue.cc:237] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.072850 18330 tablet_bootstrap.cc:492] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb: No bootstrap required, opened a new log
I20250109 20:58:01.073369 18330 ts_tablet_manager.cc:1397] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb: Time spent bootstrapping tablet: real 0.181s user 0.044s sys 0.125s
I20250109 20:58:01.074337 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9942465bba148559c8310fecb210d09" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe"
I20250109 20:58:01.074939 18210 raft_consensus.cc:3054] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.076393 18330 raft_consensus.cc:357] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.077222 18330 raft_consensus.cc:383] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:01.077591 18330 raft_consensus.cc:738] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ca066a3a647558af06e1ddf72cfcb, State: Initialized, Role: FOLLOWER
I20250109 20:58:01.078373 18330 consensus_queue.cc:260] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.079156 18335 raft_consensus.cc:2743] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:58:01.080024 18332 tablet_bootstrap.cc:492] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe: No bootstrap required, opened a new log
I20250109 20:58:01.080540 18332 ts_tablet_manager.cc:1397] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe: Time spent bootstrapping tablet: real 0.187s user 0.056s sys 0.100s
I20250109 20:58:01.080868 18356 raft_consensus.cc:695] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Becoming Leader. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Running, Role: LEADER
I20250109 20:58:01.081741 18356 consensus_queue.cc:237] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.083395 18332 raft_consensus.cc:357] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.087064 18210 raft_consensus.cc:2463] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 1.
I20250109 20:58:01.084532 18332 raft_consensus.cc:383] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:01.087519 18332 raft_consensus.cc:738] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c1b06aef574962914c99b98dcbd5fe, State: Initialized, Role: FOLLOWER
I20250109 20:58:01.088409 18250 leader_election.cc:304] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c33d4d09803d48f99f5f4bf8ac33d402; no voters: c84ca066a3a647558af06e1ddf72cfcb
I20250109 20:58:01.089926 18369 raft_consensus.cc:2798] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:01.088918 18332 consensus_queue.cc:260] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.091048 18369 raft_consensus.cc:695] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Becoming Leader. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Running, Role: LEADER
I20250109 20:58:01.092437 18369 consensus_queue.cc:237] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.093159 18332 ts_tablet_manager.cc:1428] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250109 20:58:01.091804 18331 ts_tablet_manager.cc:1428] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402: Time spent starting tablet: real 0.059s user 0.017s sys 0.012s
I20250109 20:58:01.095340 18332 tablet_bootstrap.cc:492] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe: Bootstrap starting.
I20250109 20:58:01.095341 18331 tablet_bootstrap.cc:492] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402: Bootstrap starting.
I20250109 20:58:01.097084 18330 ts_tablet_manager.cc:1428] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb: Time spent starting tablet: real 0.023s user 0.006s sys 0.007s
I20250109 20:58:01.101047 18330 tablet_bootstrap.cc:492] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb: Bootstrap starting.
I20250109 20:58:01.116091 18332 tablet_bootstrap.cc:654] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:01.117354 18330 tablet_bootstrap.cc:654] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:01.117854 18033 catalog_manager.cc:5526] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 reported cstate change: term changed from 0 to 1, leader changed from <none> to c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131). New cstate: current_term: 1 leader_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } health_report { overall_health: HEALTHY } } }
I20250109 20:58:01.122155 18331 tablet_bootstrap.cc:654] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:01.126837 18332 tablet_bootstrap.cc:492] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe: No bootstrap required, opened a new log
I20250109 20:58:01.127279 18332 ts_tablet_manager.cc:1397] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.002s
I20250109 20:58:01.127343 18330 tablet_bootstrap.cc:492] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb: No bootstrap required, opened a new log
I20250109 20:58:01.128145 18330 ts_tablet_manager.cc:1397] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.009s
I20250109 20:58:01.130070 18332 raft_consensus.cc:357] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.130839 18332 raft_consensus.cc:383] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:01.131124 18332 raft_consensus.cc:738] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c1b06aef574962914c99b98dcbd5fe, State: Initialized, Role: FOLLOWER
I20250109 20:58:01.131785 18332 consensus_queue.cc:260] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.131909 18330 raft_consensus.cc:357] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.132853 18330 raft_consensus.cc:383] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:01.133188 18330 raft_consensus.cc:738] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ca066a3a647558af06e1ddf72cfcb, State: Initialized, Role: FOLLOWER
I20250109 20:58:01.134327 18331 tablet_bootstrap.cc:492] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402: No bootstrap required, opened a new log
I20250109 20:58:01.134402 18332 ts_tablet_manager.cc:1428] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250109 20:58:01.134030 18330 consensus_queue.cc:260] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.134804 18331 ts_tablet_manager.cc:1397] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.007s
I20250109 20:58:01.135957 18332 tablet_bootstrap.cc:492] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe: Bootstrap starting.
I20250109 20:58:01.137274 18331 raft_consensus.cc:357] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.137979 18331 raft_consensus.cc:383] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:01.138269 18331 raft_consensus.cc:738] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Initialized, Role: FOLLOWER
I20250109 20:58:01.139503 18331 consensus_queue.cc:260] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.141681 18330 ts_tablet_manager.cc:1428] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb: Time spent starting tablet: real 0.013s user 0.006s sys 0.002s
I20250109 20:58:01.142333 18331 ts_tablet_manager.cc:1428] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250109 20:58:01.142813 18330 tablet_bootstrap.cc:492] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb: Bootstrap starting.
I20250109 20:58:01.142807 18332 tablet_bootstrap.cc:654] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:01.149109 18330 tablet_bootstrap.cc:654] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:01.151130 18332 tablet_bootstrap.cc:492] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe: No bootstrap required, opened a new log
I20250109 20:58:01.150274 18034 catalog_manager.cc:5526] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 reported cstate change: term changed from 0 to 1, leader changed from <none> to c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131). New cstate: current_term: 1 leader_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } health_report { overall_health: HEALTHY } } }
I20250109 20:58:01.151486 18332 ts_tablet_manager.cc:1397] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250109 20:58:01.152215 18034 catalog_manager.cc:5526] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 reported cstate change: term changed from 0 to 1, leader changed from <none> to c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131). New cstate: current_term: 1 leader_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } health_report { overall_health: HEALTHY } } }
I20250109 20:58:01.153705 18332 raft_consensus.cc:357] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.154368 18330 tablet_bootstrap.cc:492] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb: No bootstrap required, opened a new log
I20250109 20:58:01.154335 18332 raft_consensus.cc:383] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:01.154811 18332 raft_consensus.cc:738] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c1b06aef574962914c99b98dcbd5fe, State: Initialized, Role: FOLLOWER
I20250109 20:58:01.154893 18330 ts_tablet_manager.cc:1397] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250109 20:58:01.155500 18332 consensus_queue.cc:260] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.157505 18332 ts_tablet_manager.cc:1428] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250109 20:58:01.157359 18330 raft_consensus.cc:357] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.158035 18330 raft_consensus.cc:383] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:01.158288 18330 raft_consensus.cc:738] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ca066a3a647558af06e1ddf72cfcb, State: Initialized, Role: FOLLOWER
I20250109 20:58:01.158828 18330 consensus_queue.cc:260] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.160733 18330 ts_tablet_manager.cc:1428] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250109 20:58:01.182042 18369 raft_consensus.cc:491] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:01.182544 18369 raft_consensus.cc:513] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:01.184221 18369 leader_election.cc:290] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:01.184942 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec5186b23f94af985e2ede96c30b0dc" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb" is_pre_election: true
I20250109 20:58:01.185096 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec5186b23f94af985e2ede96c30b0dc" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe" is_pre_election: true
I20250109 20:58:01.185492 18136 raft_consensus.cc:2463] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 0.
I20250109 20:58:01.185648 18340 raft_consensus.cc:491] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:01.185696 18210 raft_consensus.cc:2463] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 0.
I20250109 20:58:01.186343 18248 leader_election.cc:304] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c33d4d09803d48f99f5f4bf8ac33d402, c84ca066a3a647558af06e1ddf72cfcb; no voters:
I20250109 20:58:01.186235 18340 raft_consensus.cc:513] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.187033 18369 raft_consensus.cc:2798] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:58:01.187331 18369 raft_consensus.cc:491] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:58:01.187566 18369 raft_consensus.cc:3054] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.187976 18340 leader_election.cc:290] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:01.188747 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f8261bc37f4e8ea89f4c02dc3c4478" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb" is_pre_election: true
I20250109 20:58:01.189258 18136 raft_consensus.cc:2463] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45c1b06aef574962914c99b98dcbd5fe in term 0.
I20250109 20:58:01.189152 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f8261bc37f4e8ea89f4c02dc3c4478" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" is_pre_election: true
I20250109 20:58:01.189905 18285 raft_consensus.cc:2463] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45c1b06aef574962914c99b98dcbd5fe in term 0.
I20250109 20:58:01.190400 18174 leader_election.cc:304] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c84ca066a3a647558af06e1ddf72cfcb; no voters:
I20250109 20:58:01.191151 18340 raft_consensus.cc:2798] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:58:01.191478 18340 raft_consensus.cc:491] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:58:01.191725 18340 raft_consensus.cc:3054] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.193150 18369 raft_consensus.cc:513] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:01.194664 18369 leader_election.cc:290] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Requested vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:01.195286 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec5186b23f94af985e2ede96c30b0dc" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
I20250109 20:58:01.195818 18136 raft_consensus.cc:3054] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.196079 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec5186b23f94af985e2ede96c30b0dc" candidate_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe"
I20250109 20:58:01.196650 18210 raft_consensus.cc:3054] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.196591 18370 raft_consensus.cc:491] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:01.196704 18340 raft_consensus.cc:513] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.197355 18370 raft_consensus.cc:513] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.198625 18340 leader_election.cc:290] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 1 election: Requested vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:01.199249 18370 leader_election.cc:290] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803), c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:01.200438 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9942465bba148559c8310fecb210d09" candidate_uuid: "c84ca066a3a647558af06e1ddf72cfcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c1b06aef574962914c99b98dcbd5fe" is_pre_election: true
I20250109 20:58:01.201114 18211 raft_consensus.cc:2388] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c84ca066a3a647558af06e1ddf72cfcb in current term 1: Already voted for candidate c33d4d09803d48f99f5f4bf8ac33d402 in this term.
I20250109 20:58:01.201676 18135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f8261bc37f4e8ea89f4c02dc3c4478" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
I20250109 20:58:01.202358 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f8261bc37f4e8ea89f4c02dc3c4478" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402"
I20250109 20:58:01.202422 18135 raft_consensus.cc:3054] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.202848 18284 raft_consensus.cc:3054] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.200891 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9942465bba148559c8310fecb210d09" candidate_uuid: "c84ca066a3a647558af06e1ddf72cfcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" is_pre_election: true
I20250109 20:58:01.206784 18101 leader_election.cc:304] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c84ca066a3a647558af06e1ddf72cfcb; no voters: 45c1b06aef574962914c99b98dcbd5fe, c33d4d09803d48f99f5f4bf8ac33d402
I20250109 20:58:01.207068 18136 raft_consensus.cc:2463] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 1.
I20250109 20:58:01.207960 18370 raft_consensus.cc:3054] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.207998 18248 leader_election.cc:304] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c33d4d09803d48f99f5f4bf8ac33d402, c84ca066a3a647558af06e1ddf72cfcb; no voters:
I20250109 20:58:01.208922 18369 raft_consensus.cc:2798] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:01.209375 18369 raft_consensus.cc:695] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Becoming Leader. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Running, Role: LEADER
I20250109 20:58:01.209735 18210 raft_consensus.cc:2463] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33d4d09803d48f99f5f4bf8ac33d402 in term 1.
I20250109 20:58:01.210309 18369 consensus_queue.cc:237] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:01.212271 18284 raft_consensus.cc:2463] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45c1b06aef574962914c99b98dcbd5fe in term 1.
I20250109 20:58:01.214844 18176 leader_election.cc:304] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c33d4d09803d48f99f5f4bf8ac33d402; no voters:
I20250109 20:58:01.215845 18135 raft_consensus.cc:2463] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45c1b06aef574962914c99b98dcbd5fe in term 1.
I20250109 20:58:01.216162 18370 raft_consensus.cc:2743] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:58:01.216368 18340 raft_consensus.cc:2798] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:01.217859 18340 raft_consensus.cc:695] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 1 LEADER]: Becoming Leader. State: Replica: 45c1b06aef574962914c99b98dcbd5fe, State: Running, Role: LEADER
I20250109 20:58:01.218662 18340 consensus_queue.cc:237] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.222390 18033 catalog_manager.cc:5526] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 reported cstate change: term changed from 0 to 1, leader changed from <none> to c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131). New cstate: current_term: 1 leader_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20250109 20:58:01.226086 18032 catalog_manager.cc:5526] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130). New cstate: current_term: 1 leader_uuid: "45c1b06aef574962914c99b98dcbd5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } health_report { overall_health: UNKNOWN } } }
I20250109 20:58:01.264186 18340 raft_consensus.cc:491] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:01.264583 18340 raft_consensus.cc:513] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.266135 18340 leader_election.cc:290] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:01.266914 18135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7500274b7719441596814acc45fa3028" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb" is_pre_election: true
I20250109 20:58:01.267110 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7500274b7719441596814acc45fa3028" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" is_pre_election: true
I20250109 20:58:01.267539 18135 raft_consensus.cc:2388] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45c1b06aef574962914c99b98dcbd5fe in current term 1: Already voted for candidate c33d4d09803d48f99f5f4bf8ac33d402 in this term.
I20250109 20:58:01.268406 18174 leader_election.cc:304] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe; no voters: c33d4d09803d48f99f5f4bf8ac33d402, c84ca066a3a647558af06e1ddf72cfcb
I20250109 20:58:01.269009 18340 raft_consensus.cc:3054] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.273032 18340 raft_consensus.cc:2743] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:58:01.351890 18369 consensus_queue.cc:1035] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.362406 18369 consensus_queue.cc:1035] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.377159 18369 consensus_queue.cc:1035] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.385957 18369 consensus_queue.cc:1035] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.476832 18369 consensus_queue.cc:1035] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.486598 18369 consensus_queue.cc:1035] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.508595 18340 raft_consensus.cc:491] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:01.508958 18340 raft_consensus.cc:513] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.510311 18340 leader_election.cc:290] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:01.510882 18135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6f9e2410d440079d9ff8e622c7369a" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb" is_pre_election: true
I20250109 20:58:01.511215 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6f9e2410d440079d9ff8e622c7369a" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" is_pre_election: true
I20250109 20:58:01.511502 18135 raft_consensus.cc:2463] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45c1b06aef574962914c99b98dcbd5fe in term 0.
I20250109 20:58:01.511837 18284 raft_consensus.cc:2463] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45c1b06aef574962914c99b98dcbd5fe in term 0.
I20250109 20:58:01.512440 18174 leader_election.cc:304] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c84ca066a3a647558af06e1ddf72cfcb; no voters:
I20250109 20:58:01.513025 18340 raft_consensus.cc:2798] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:58:01.513334 18340 raft_consensus.cc:491] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:58:01.513561 18340 raft_consensus.cc:3054] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.517297 18340 raft_consensus.cc:513] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.518704 18340 leader_election.cc:290] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 1 election: Requested vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:01.519244 18135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6f9e2410d440079d9ff8e622c7369a" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
I20250109 20:58:01.519565 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6f9e2410d440079d9ff8e622c7369a" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402"
I20250109 20:58:01.519771 18135 raft_consensus.cc:3054] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.520054 18284 raft_consensus.cc:3054] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:01.525750 18284 raft_consensus.cc:2463] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45c1b06aef574962914c99b98dcbd5fe in term 1.
I20250109 20:58:01.525750 18135 raft_consensus.cc:2463] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45c1b06aef574962914c99b98dcbd5fe in term 1.
I20250109 20:58:01.526842 18176 leader_election.cc:304] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c33d4d09803d48f99f5f4bf8ac33d402; no voters:
I20250109 20:58:01.527483 18340 raft_consensus.cc:2798] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:01.527813 18340 raft_consensus.cc:695] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 1 LEADER]: Becoming Leader. State: Replica: 45c1b06aef574962914c99b98dcbd5fe, State: Running, Role: LEADER
I20250109 20:58:01.528333 18340 consensus_queue.cc:237] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:01.534698 18033 catalog_manager.cc:5526] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130). New cstate: current_term: 1 leader_uuid: "45c1b06aef574962914c99b98dcbd5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } health_report { overall_health: UNKNOWN } } }
I20250109 20:58:01.562494 18369 consensus_queue.cc:1035] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.568886 18356 consensus_queue.cc:1035] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.572948 18369 consensus_queue.cc:1035] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.587602 18356 consensus_queue.cc:1035] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.592590 18369 consensus_queue.cc:1035] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.603502 18368 consensus_queue.cc:1035] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.608892 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:01.616667 18398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:01.620344 18401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:01.619560 18399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:01.623612 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:01.624814 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:01.625104 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:01.625268 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456281625252 us; error 0 us; skew 500 ppm
I20250109 20:58:01.625913 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:01.628463 17466 webserver.cc:458] Webserver started at http://127.17.14.132:43955/ using document root <none> and password file <none>
I20250109 20:58:01.628880 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:01.629051 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:01.629293 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:01.630529 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-3-root/instance:
uuid: "f5b05b0a86c1457f8a1d0debf65cecb4"
format_stamp: "Formatted at 2025-01-09 20:58:01 on dist-test-slave-j54z"
I20250109 20:58:01.634882 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20250109 20:58:01.637853 18406 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:01.638594 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20250109 20:58:01.638828 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-3-root
uuid: "f5b05b0a86c1457f8a1d0debf65cecb4"
format_stamp: "Formatted at 2025-01-09 20:58:01 on dist-test-slave-j54z"
I20250109 20:58:01.639084 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736456270445890-17466-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:01.648707 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:01.649971 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:01.651110 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:58:01.653239 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:58:01.653409 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:01.653651 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:58:01.653801 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:01.692401 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.132:41789
I20250109 20:58:01.692445 18468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.132:41789 every 8 connection(s)
I20250109 20:58:01.705703 18469 heartbeater.cc:346] Connected to a master server at 127.17.14.190:45483
I20250109 20:58:01.706040 18469 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:01.706718 18469 heartbeater.cc:510] Master 127.17.14.190:45483 requested a full tablet report, sending...
I20250109 20:58:01.708321 18034 ts_manager.cc:194] Registered new tserver with Master: f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132:41789)
I20250109 20:58:01.709674 18034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43302
I20250109 20:58:01.757189 18340 consensus_queue.cc:1035] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:01.769105 18474 consensus_queue.cc:1035] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:02.003151 18474 consensus_queue.cc:1035] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:02.012698 18474 consensus_queue.cc:1035] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:02.711977 18469 heartbeater.cc:502] Master 127.17.14.190:45483 was elected leader, sending a full tablet report...
I20250109 20:58:02.919849 18087 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:02.936836 18284 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "40bb82b56471437eba54b55ebb6c20ce"
dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402"
mode: GRACEFUL
new_leader_uuid: "45c1b06aef574962914c99b98dcbd5fe"
from {username='slave'} at 127.0.0.1:34350
I20250109 20:58:02.937469 18284 raft_consensus.cc:604] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Received request to transfer leadership to TS 45c1b06aef574962914c99b98dcbd5fe
I20250109 20:58:02.939894 18284 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7500274b7719441596814acc45fa3028"
dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402"
mode: GRACEFUL
new_leader_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
from {username='slave'} at 127.0.0.1:34350
I20250109 20:58:02.940462 18284 raft_consensus.cc:604] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Received request to transfer leadership to TS c84ca066a3a647558af06e1ddf72cfcb
I20250109 20:58:02.942631 18284 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "762caa2bc7564c9d88448466f319a2bf"
dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402"
mode: GRACEFUL
new_leader_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
from {username='slave'} at 127.0.0.1:34350
I20250109 20:58:02.943113 18284 raft_consensus.cc:604] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Received request to transfer leadership to TS c84ca066a3a647558af06e1ddf72cfcb
I20250109 20:58:02.945088 18284 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bec5186b23f94af985e2ede96c30b0dc"
dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402"
mode: GRACEFUL
new_leader_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
from {username='slave'} at 127.0.0.1:34350
I20250109 20:58:02.945565 18284 raft_consensus.cc:604] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Received request to transfer leadership to TS c84ca066a3a647558af06e1ddf72cfcb
I20250109 20:58:02.946431 18087 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250109 20:58:02.946903 18087 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:03.079999 18480 raft_consensus.cc:988] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402: : Instructing follower c84ca066a3a647558af06e1ddf72cfcb to start an election
I20250109 20:58:03.080461 18369 raft_consensus.cc:1076] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Signalling peer c84ca066a3a647558af06e1ddf72cfcb to start an election
I20250109 20:58:03.082027 18136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7500274b7719441596814acc45fa3028"
dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
from {username='slave'} at 127.0.0.1:53272
I20250109 20:58:03.082556 18136 raft_consensus.cc:491] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250109 20:58:03.082845 18136 raft_consensus.cc:3054] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:03.087158 18136 raft_consensus.cc:513] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:03.088572 18136 leader_election.cc:290] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 2 election: Requested vote from peers 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803), c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:03.089394 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7500274b7719441596814acc45fa3028" candidate_uuid: "c84ca066a3a647558af06e1ddf72cfcb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "45c1b06aef574962914c99b98dcbd5fe"
I20250109 20:58:03.089643 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7500274b7719441596814acc45fa3028" candidate_uuid: "c84ca066a3a647558af06e1ddf72cfcb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402"
I20250109 20:58:03.090006 18210 raft_consensus.cc:3054] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:03.090201 18284 raft_consensus.cc:3049] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Stepping down as leader of term 1
I20250109 20:58:03.090507 18284 raft_consensus.cc:738] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Running, Role: LEADER
I20250109 20:58:03.091202 18284 consensus_queue.cc:260] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:03.091989 18284 raft_consensus.cc:3054] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:03.094569 18210 raft_consensus.cc:2463] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84ca066a3a647558af06e1ddf72cfcb in term 2.
I20250109 20:58:03.095206 18211 raft_consensus.cc:1235] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Rejecting Update request from peer c33d4d09803d48f99f5f4bf8ac33d402 for earlier term 1. Current term is 2. Ops: []
I20250109 20:58:03.095821 18101 leader_election.cc:304] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c84ca066a3a647558af06e1ddf72cfcb; no voters:
I20250109 20:58:03.097400 18284 raft_consensus.cc:2463] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84ca066a3a647558af06e1ddf72cfcb in term 2.
I20250109 20:58:03.097982 18481 raft_consensus.cc:2798] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Leader election won for term 2
I20250109 20:58:03.099138 18481 raft_consensus.cc:695] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 2 LEADER]: Becoming Leader. State: Replica: c84ca066a3a647558af06e1ddf72cfcb, State: Running, Role: LEADER
I20250109 20:58:03.100173 18481 consensus_queue.cc:237] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:03.106709 18033 catalog_manager.cc:5526] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb reported cstate change: term changed from 1 to 2, leader changed from c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131) to c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129). New cstate: current_term: 2 leader_uuid: "c84ca066a3a647558af06e1ddf72cfcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } health_report { overall_health: UNKNOWN } } }
I20250109 20:58:03.358120 18369 raft_consensus.cc:988] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402: : Instructing follower c84ca066a3a647558af06e1ddf72cfcb to start an election
I20250109 20:58:03.358510 18480 raft_consensus.cc:1076] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Signalling peer c84ca066a3a647558af06e1ddf72cfcb to start an election
I20250109 20:58:03.359791 18136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bec5186b23f94af985e2ede96c30b0dc"
dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
from {username='slave'} at 127.0.0.1:53272
I20250109 20:58:03.360261 18136 raft_consensus.cc:491] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250109 20:58:03.360548 18136 raft_consensus.cc:3054] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:03.364672 18136 raft_consensus.cc:513] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:03.366225 18136 leader_election.cc:290] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 2 election: Requested vote from peers c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533), 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803)
I20250109 20:58:03.367045 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec5186b23f94af985e2ede96c30b0dc" candidate_uuid: "c84ca066a3a647558af06e1ddf72cfcb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402"
I20250109 20:58:03.367319 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec5186b23f94af985e2ede96c30b0dc" candidate_uuid: "c84ca066a3a647558af06e1ddf72cfcb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "45c1b06aef574962914c99b98dcbd5fe"
I20250109 20:58:03.367556 18284 raft_consensus.cc:3049] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Stepping down as leader of term 1
I20250109 20:58:03.367822 18211 raft_consensus.cc:3054] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:03.367871 18284 raft_consensus.cc:738] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Running, Role: LEADER
I20250109 20:58:03.368659 18284 consensus_queue.cc:260] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:03.369601 18284 raft_consensus.cc:3054] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:03.372414 18211 raft_consensus.cc:2463] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84ca066a3a647558af06e1ddf72cfcb in term 2.
I20250109 20:58:03.373315 18101 leader_election.cc:304] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c84ca066a3a647558af06e1ddf72cfcb; no voters:
I20250109 20:58:03.373890 18284 raft_consensus.cc:2463] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84ca066a3a647558af06e1ddf72cfcb in term 2.
I20250109 20:58:03.374004 18481 raft_consensus.cc:2798] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Leader election won for term 2
I20250109 20:58:03.374475 18481 raft_consensus.cc:695] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 2 LEADER]: Becoming Leader. State: Replica: c84ca066a3a647558af06e1ddf72cfcb, State: Running, Role: LEADER
I20250109 20:58:03.375238 18481 consensus_queue.cc:237] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } }
I20250109 20:58:03.381456 18033 catalog_manager.cc:5526] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb reported cstate change: term changed from 1 to 2, leader changed from c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131) to c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129). New cstate: current_term: 2 leader_uuid: "c84ca066a3a647558af06e1ddf72cfcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20250109 20:58:03.395187 18369 raft_consensus.cc:988] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402: : Instructing follower 45c1b06aef574962914c99b98dcbd5fe to start an election
I20250109 20:58:03.395576 18480 raft_consensus.cc:1076] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Signalling peer 45c1b06aef574962914c99b98dcbd5fe to start an election
I20250109 20:58:03.396900 18211 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40bb82b56471437eba54b55ebb6c20ce"
dest_uuid: "45c1b06aef574962914c99b98dcbd5fe"
from {username='slave'} at 127.0.0.1:54850
I20250109 20:58:03.397354 18211 raft_consensus.cc:491] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250109 20:58:03.397687 18211 raft_consensus.cc:3054] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:03.402257 18211 raft_consensus.cc:513] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:03.403753 18211 leader_election.cc:290] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 2 election: Requested vote from peers c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553), c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:03.404567 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bb82b56471437eba54b55ebb6c20ce" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
I20250109 20:58:03.404852 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bb82b56471437eba54b55ebb6c20ce" candidate_uuid: "45c1b06aef574962914c99b98dcbd5fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402"
I20250109 20:58:03.405153 18136 raft_consensus.cc:3054] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:03.405458 18284 raft_consensus.cc:3049] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Stepping down as leader of term 1
I20250109 20:58:03.405751 18284 raft_consensus.cc:738] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Running, Role: LEADER
I20250109 20:58:03.406337 18284 consensus_queue.cc:260] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:03.407191 18284 raft_consensus.cc:3054] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:03.409617 18136 raft_consensus.cc:2463] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45c1b06aef574962914c99b98dcbd5fe in term 2.
I20250109 20:58:03.410621 18174 leader_election.cc:304] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c84ca066a3a647558af06e1ddf72cfcb; no voters:
I20250109 20:58:03.411314 18380 raft_consensus.cc:2798] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Leader election won for term 2
I20250109 20:58:03.412441 18284 raft_consensus.cc:2463] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45c1b06aef574962914c99b98dcbd5fe in term 2.
I20250109 20:58:03.412470 18380 raft_consensus.cc:695] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 2 LEADER]: Becoming Leader. State: Replica: 45c1b06aef574962914c99b98dcbd5fe, State: Running, Role: LEADER
I20250109 20:58:03.413462 18380 consensus_queue.cc:237] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:03.419562 18033 catalog_manager.cc:5526] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe reported cstate change: term changed from 1 to 2, leader changed from c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131) to 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130). New cstate: current_term: 2 leader_uuid: "45c1b06aef574962914c99b98dcbd5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } health_report { overall_health: UNKNOWN } } }
I20250109 20:58:03.448625 18369 raft_consensus.cc:988] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402: : Instructing follower c84ca066a3a647558af06e1ddf72cfcb to start an election
I20250109 20:58:03.449012 18480 raft_consensus.cc:1076] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Signalling peer c84ca066a3a647558af06e1ddf72cfcb to start an election
I20250109 20:58:03.450269 18136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "762caa2bc7564c9d88448466f319a2bf"
dest_uuid: "c84ca066a3a647558af06e1ddf72cfcb"
from {username='slave'} at 127.0.0.1:53272
I20250109 20:58:03.450686 18136 raft_consensus.cc:491] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250109 20:58:03.450958 18136 raft_consensus.cc:3054] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:03.455132 18136 raft_consensus.cc:513] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:03.456508 18136 leader_election.cc:290] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 2 election: Requested vote from peers 45c1b06aef574962914c99b98dcbd5fe (127.17.14.130:36803), c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:03.457330 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762caa2bc7564c9d88448466f319a2bf" candidate_uuid: "c84ca066a3a647558af06e1ddf72cfcb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "45c1b06aef574962914c99b98dcbd5fe"
I20250109 20:58:03.457674 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762caa2bc7564c9d88448466f319a2bf" candidate_uuid: "c84ca066a3a647558af06e1ddf72cfcb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c33d4d09803d48f99f5f4bf8ac33d402"
I20250109 20:58:03.457811 18211 raft_consensus.cc:3054] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:03.458192 18284 raft_consensus.cc:3049] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Stepping down as leader of term 1
I20250109 20:58:03.458479 18284 raft_consensus.cc:738] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c33d4d09803d48f99f5f4bf8ac33d402, State: Running, Role: LEADER
I20250109 20:58:03.459122 18284 consensus_queue.cc:260] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:03.460034 18284 raft_consensus.cc:3054] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:03.462205 18211 raft_consensus.cc:2463] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84ca066a3a647558af06e1ddf72cfcb in term 2.
I20250109 20:58:03.463055 18101 leader_election.cc:304] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c1b06aef574962914c99b98dcbd5fe, c84ca066a3a647558af06e1ddf72cfcb; no voters:
I20250109 20:58:03.463660 18481 raft_consensus.cc:2798] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Leader election won for term 2
I20250109 20:58:03.464020 18481 raft_consensus.cc:695] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 2 LEADER]: Becoming Leader. State: Replica: c84ca066a3a647558af06e1ddf72cfcb, State: Running, Role: LEADER
I20250109 20:58:03.464252 18284 raft_consensus.cc:2463] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84ca066a3a647558af06e1ddf72cfcb in term 2.
I20250109 20:58:03.464766 18481 consensus_queue.cc:237] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } }
I20250109 20:58:03.471199 18033 catalog_manager.cc:5526] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb reported cstate change: term changed from 1 to 2, leader changed from c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131) to c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129). New cstate: current_term: 2 leader_uuid: "c84ca066a3a647558af06e1ddf72cfcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } health_report { overall_health: UNKNOWN } } }
I20250109 20:58:03.583963 18284 raft_consensus.cc:1270] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Refusing update from remote peer c84ca066a3a647558af06e1ddf72cfcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:58:03.585103 18481 consensus_queue.cc:1035] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:03.593544 18211 raft_consensus.cc:1270] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Refusing update from remote peer c84ca066a3a647558af06e1ddf72cfcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:58:03.595003 18481 consensus_queue.cc:1035] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:03.833868 18284 raft_consensus.cc:1270] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Refusing update from remote peer 45c1b06aef574962914c99b98dcbd5fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:58:03.834932 18380 consensus_queue.cc:1035] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:03.842883 18136 raft_consensus.cc:1270] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Refusing update from remote peer 45c1b06aef574962914c99b98dcbd5fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:58:03.844393 18498 consensus_queue.cc:1035] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250109 20:58:03.969080 18211 raft_consensus.cc:1270] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Refusing update from remote peer c84ca066a3a647558af06e1ddf72cfcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:58:03.970486 18481 consensus_queue.cc:1035] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:03.971060 18284 raft_consensus.cc:1270] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Refusing update from remote peer c84ca066a3a647558af06e1ddf72cfcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:58:03.972199 18481 consensus_queue.cc:1035] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:04.010424 18211 raft_consensus.cc:1270] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Refusing update from remote peer c84ca066a3a647558af06e1ddf72cfcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:58:04.011478 18502 consensus_queue.cc:1035] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:04.018909 18284 raft_consensus.cc:1270] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Refusing update from remote peer c84ca066a3a647558af06e1ddf72cfcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:58:04.020570 18502 consensus_queue.cc:1035] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:04.947729 18087 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:04.952999 18087 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:04.953408 18087 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:06.954414 18087 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:06.954792 18136 consensus_queue.cc:237] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } }
I20250109 20:58:06.963732 18211 raft_consensus.cc:1270] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Refusing update from remote peer c84ca066a3a647558af06e1ddf72cfcb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 20:58:06.965870 18481 consensus_queue.cc:1035] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250109 20:58:06.970526 18284 raft_consensus.cc:1270] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Refusing update from remote peer c84ca066a3a647558af06e1ddf72cfcb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 20:58:06.970814 18087 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:06.971421 18087 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:06.972149 18512 consensus_queue.cc:1035] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 20:58:06.976012 18512 raft_consensus.cc:2949] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:06.978734 18211 raft_consensus.cc:2949] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
W20250109 20:58:06.981331 18102 consensus_peers.cc:487] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb -> Peer f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132:41789): Couldn't send request to peer f5b05b0a86c1457f8a1d0debf65cecb4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bec5186b23f94af985e2ede96c30b0dc. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:58:06.984062 18284 raft_consensus.cc:2949] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.082517 18034 catalog_manager.cc:5526] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe reported cstate change: config changed from index -1 to 3, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New cstate: current_term: 2 leader_uuid: "c84ca066a3a647558af06e1ddf72cfcb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.095124 18284 consensus_queue.cc:237] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } }
I20250109 20:58:07.102658 18211 raft_consensus.cc:1270] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Refusing update from remote peer c33d4d09803d48f99f5f4bf8ac33d402: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:58:07.104333 18136 raft_consensus.cc:1270] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Refusing update from remote peer c33d4d09803d48f99f5f4bf8ac33d402: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:58:07.104218 18523 consensus_queue.cc:1035] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:07.105715 18480 consensus_queue.cc:1035] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:07.113689 18523 raft_consensus.cc:2949] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
W20250109 20:58:07.114725 18251 consensus_peers.cc:487] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 -> Peer f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132:41789): Couldn't send request to peer f5b05b0a86c1457f8a1d0debf65cecb4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9942465bba148559c8310fecb210d09. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:58:07.115320 18211 raft_consensus.cc:2949] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.117386 18136 raft_consensus.cc:2949] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.190503 18033 catalog_manager.cc:5526] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe reported cstate change: config changed from index -1 to 2, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New cstate: current_term: 1 leader_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.193564 18136 consensus_queue.cc:237] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } attrs { replace: true } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } }
I20250109 20:58:07.203691 18284 raft_consensus.cc:1270] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Refusing update from remote peer c84ca066a3a647558af06e1ddf72cfcb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250109 20:58:07.204319 18102 consensus_peers.cc:487] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb -> Peer f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132:41789): Couldn't send request to peer f5b05b0a86c1457f8a1d0debf65cecb4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7500274b7719441596814acc45fa3028. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:58:07.204504 18211 raft_consensus.cc:1270] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Refusing update from remote peer c84ca066a3a647558af06e1ddf72cfcb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 20:58:07.205207 18512 consensus_queue.cc:1035] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 20:58:07.206092 18481 consensus_queue.cc:1035] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 20:58:07.217239 18532 raft_consensus.cc:2949] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } attrs { replace: true } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.218542 18284 raft_consensus.cc:2949] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } attrs { replace: true } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.218928 18211 raft_consensus.cc:2949] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } attrs { replace: true } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.290827 18034 catalog_manager.cc:5526] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 reported cstate change: config changed from index -1 to 3, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New cstate: current_term: 2 leader_uuid: "c84ca066a3a647558af06e1ddf72cfcb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } attrs { replace: true } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.292980 18284 consensus_queue.cc:237] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } }
I20250109 20:58:07.299645 18211 raft_consensus.cc:1270] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Refusing update from remote peer c33d4d09803d48f99f5f4bf8ac33d402: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250109 20:58:07.300034 18251 consensus_peers.cc:487] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 -> Peer f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132:41789): Couldn't send request to peer f5b05b0a86c1457f8a1d0debf65cecb4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bd2b37a3e154ffea27c42bd78441e4a. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:58:07.301126 18523 consensus_queue.cc:1035] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:07.305791 18136 raft_consensus.cc:1270] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Refusing update from remote peer c33d4d09803d48f99f5f4bf8ac33d402: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:58:07.307262 18480 consensus_queue.cc:1035] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:07.309157 18527 raft_consensus.cc:2949] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.313079 18211 raft_consensus.cc:2949] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.319790 18136 raft_consensus.cc:2949] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.390316 18032 catalog_manager.cc:5526] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb reported cstate change: config changed from index -1 to 2, NON_VOTER f5b05b0a86c1457f8a1d0debf65cecb4 (127.17.14.132) added. New cstate: current_term: 1 leader_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } } }
I20250109 20:58:07.501166 18536 ts_tablet_manager.cc:927] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4: Initiating tablet copy from peer c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553)
I20250109 20:58:07.502846 18536 tablet_copy_client.cc:323] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.129:44553
I20250109 20:58:07.513757 18146 tablet_copy_service.cc:140] P c84ca066a3a647558af06e1ddf72cfcb: Received BeginTabletCopySession request for tablet bec5186b23f94af985e2ede96c30b0dc from peer f5b05b0a86c1457f8a1d0debf65cecb4 ({username='slave'} at 127.0.0.1:39956)
I20250109 20:58:07.514434 18146 tablet_copy_service.cc:161] P c84ca066a3a647558af06e1ddf72cfcb: Beginning new tablet copy session on tablet bec5186b23f94af985e2ede96c30b0dc from peer f5b05b0a86c1457f8a1d0debf65cecb4 at {username='slave'} at 127.0.0.1:39956: session id = f5b05b0a86c1457f8a1d0debf65cecb4-bec5186b23f94af985e2ede96c30b0dc
I20250109 20:58:07.522200 18146 tablet_copy_source_session.cc:215] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 20:58:07.525605 18536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bec5186b23f94af985e2ede96c30b0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:07.538057 18536 tablet_copy_client.cc:806] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Starting download of 0 data blocks...
I20250109 20:58:07.539669 18536 tablet_copy_client.cc:670] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Starting download of 1 WAL segments...
I20250109 20:58:07.544740 18536 tablet_copy_client.cc:538] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 20:58:07.585295 18536 tablet_bootstrap.cc:492] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4: Bootstrap starting.
I20250109 20:58:07.606868 18536 tablet_bootstrap.cc:492] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:58:07.607743 18536 tablet_bootstrap.cc:492] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4: Bootstrap complete.
I20250109 20:58:07.608338 18536 ts_tablet_manager.cc:1397] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.004s
I20250109 20:58:07.610859 18536 raft_consensus.cc:357] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } }
I20250109 20:58:07.611631 18536 raft_consensus.cc:738] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f5b05b0a86c1457f8a1d0debf65cecb4, State: Initialized, Role: LEARNER
I20250109 20:58:07.612177 18536 consensus_queue.cc:260] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } }
I20250109 20:58:07.616580 18536 ts_tablet_manager.cc:1428] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250109 20:58:07.618243 18146 tablet_copy_service.cc:342] P c84ca066a3a647558af06e1ddf72cfcb: Request end of tablet copy session f5b05b0a86c1457f8a1d0debf65cecb4-bec5186b23f94af985e2ede96c30b0dc received from {username='slave'} at 127.0.0.1:39956
I20250109 20:58:07.618733 18146 tablet_copy_service.cc:434] P c84ca066a3a647558af06e1ddf72cfcb: ending tablet copy session f5b05b0a86c1457f8a1d0debf65cecb4-bec5186b23f94af985e2ede96c30b0dc on tablet bec5186b23f94af985e2ede96c30b0dc with peer f5b05b0a86c1457f8a1d0debf65cecb4
I20250109 20:58:07.671141 18536 ts_tablet_manager.cc:927] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4: Initiating tablet copy from peer c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:07.672667 18536 tablet_copy_client.cc:323] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.131:33533
I20250109 20:58:07.682785 18295 tablet_copy_service.cc:140] P c33d4d09803d48f99f5f4bf8ac33d402: Received BeginTabletCopySession request for tablet a9942465bba148559c8310fecb210d09 from peer f5b05b0a86c1457f8a1d0debf65cecb4 ({username='slave'} at 127.0.0.1:34358)
I20250109 20:58:07.683249 18295 tablet_copy_service.cc:161] P c33d4d09803d48f99f5f4bf8ac33d402: Beginning new tablet copy session on tablet a9942465bba148559c8310fecb210d09 from peer f5b05b0a86c1457f8a1d0debf65cecb4 at {username='slave'} at 127.0.0.1:34358: session id = f5b05b0a86c1457f8a1d0debf65cecb4-a9942465bba148559c8310fecb210d09
I20250109 20:58:07.689535 18295 tablet_copy_source_session.cc:215] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 20:58:07.692377 18536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9942465bba148559c8310fecb210d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:07.703148 18536 tablet_copy_client.cc:806] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Starting download of 0 data blocks...
I20250109 20:58:07.703995 18536 tablet_copy_client.cc:670] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Starting download of 1 WAL segments...
I20250109 20:58:07.707073 18536 tablet_copy_client.cc:538] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 20:58:07.725185 17466 tablet_server.cc:178] TabletServer@127.17.14.129:0 shutting down...
W20250109 20:58:07.732235 18018 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20250109 20:58:07.737025 18082 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250109 20:58:07.737432 18082 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250109 20:58:07.746189 18082 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250109 20:58:07.747408 18082 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250109 20:58:07.755712 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:58:07.756480 17466 tablet_replica.cc:331] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb: stopping tablet replica
W20250109 20:58:07.756600 18248 consensus_peers.cc:487] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 -> Peer c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553): Couldn't send request to peer c84ca066a3a647558af06e1ddf72cfcb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:58:07.757192 17466 raft_consensus.cc:2238] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.757642 17466 raft_consensus.cc:2267] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.759277 17466 tablet_replica.cc:331] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb: stopping tablet replica
I20250109 20:58:07.759773 17466 raft_consensus.cc:2238] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.760136 17466 raft_consensus.cc:2267] T 1e6f9e2410d440079d9ff8e622c7369a P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.761739 17466 tablet_replica.cc:331] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb: stopping tablet replica
I20250109 20:58:07.762193 17466 raft_consensus.cc:2238] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.762575 17466 raft_consensus.cc:2267] T 40bb82b56471437eba54b55ebb6c20ce P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.764345 17466 tablet_replica.cc:331] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb: stopping tablet replica
I20250109 20:58:07.765017 17466 raft_consensus.cc:2238] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.765278 18546 ts_tablet_manager.cc:927] T 7500274b7719441596814acc45fa3028 P f5b05b0a86c1457f8a1d0debf65cecb4: Initiating tablet copy from peer c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553)
I20250109 20:58:07.765645 17466 raft_consensus.cc:2267] T a9942465bba148559c8310fecb210d09 P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.768172 18546 tablet_copy_client.cc:323] T 7500274b7719441596814acc45fa3028 P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.129:44553
I20250109 20:58:07.771421 17466 tablet_replica.cc:331] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb: stopping tablet replica
W20250109 20:58:07.772091 18174 consensus_peers.cc:487] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe -> Peer c84ca066a3a647558af06e1ddf72cfcb (127.17.14.129:44553): Couldn't send request to peer c84ca066a3a647558af06e1ddf72cfcb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:58:07.772228 17466 raft_consensus.cc:2238] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 2 LEADER]: Raft consensus shutting down.
I20250109 20:58:07.773183 17466 raft_consensus.cc:2267] T 7500274b7719441596814acc45fa3028 P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.775691 17466 tablet_replica.cc:331] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb: stopping tablet replica
I20250109 20:58:07.776362 17466 raft_consensus.cc:2238] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.776885 17466 raft_consensus.cc:2267] T 7bd2b37a3e154ffea27c42bd78441e4a P c84ca066a3a647558af06e1ddf72cfcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.779287 17466 tablet_replica.cc:331] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb: stopping tablet replica
I20250109 20:58:07.779924 17466 raft_consensus.cc:2238] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 2 LEADER]: Raft consensus shutting down.
I20250109 20:58:07.780866 17466 raft_consensus.cc:2267] T 762caa2bc7564c9d88448466f319a2bf P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.782820 17466 tablet_replica.cc:331] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb: stopping tablet replica
I20250109 20:58:07.783372 17466 raft_consensus.cc:2238] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 2 LEADER]: Raft consensus shutting down.
I20250109 20:58:07.784446 17466 raft_consensus.cc:2267] T bec5186b23f94af985e2ede96c30b0dc P c84ca066a3a647558af06e1ddf72cfcb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.791164 18536 tablet_bootstrap.cc:492] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4: Bootstrap starting.
I20250109 20:58:07.809131 17466 tablet_server.cc:195] TabletServer@127.17.14.129:0 shutdown complete.
I20250109 20:58:07.813629 18536 tablet_bootstrap.cc:492] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:58:07.815486 18536 tablet_bootstrap.cc:492] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4: Bootstrap complete.
I20250109 20:58:07.816109 18551 ts_tablet_manager.cc:927] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4: Initiating tablet copy from peer c33d4d09803d48f99f5f4bf8ac33d402 (127.17.14.131:33533)
I20250109 20:58:07.816169 18536 ts_tablet_manager.cc:1397] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.009s
W20250109 20:58:07.820014 18410 connection.cc:441] server connection from 127.0.0.1:56292 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:56292 (request call id 3) could send its response
I20250109 20:58:07.821465 18551 tablet_copy_client.cc:323] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.131:33533
I20250109 20:58:07.821769 18536 raft_consensus.cc:357] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } }
I20250109 20:58:07.823268 18295 tablet_copy_service.cc:140] P c33d4d09803d48f99f5f4bf8ac33d402: Received BeginTabletCopySession request for tablet 7bd2b37a3e154ffea27c42bd78441e4a from peer f5b05b0a86c1457f8a1d0debf65cecb4 ({username='slave'} at 127.0.0.1:34358)
I20250109 20:58:07.823836 18295 tablet_copy_service.cc:161] P c33d4d09803d48f99f5f4bf8ac33d402: Beginning new tablet copy session on tablet 7bd2b37a3e154ffea27c42bd78441e4a from peer f5b05b0a86c1457f8a1d0debf65cecb4 at {username='slave'} at 127.0.0.1:34358: session id = f5b05b0a86c1457f8a1d0debf65cecb4-7bd2b37a3e154ffea27c42bd78441e4a
I20250109 20:58:07.831306 18295 tablet_copy_source_session.cc:215] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 20:58:07.822803 18536 raft_consensus.cc:738] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5b05b0a86c1457f8a1d0debf65cecb4, State: Initialized, Role: LEARNER
I20250109 20:58:07.833230 18536 consensus_queue.cc:260] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } }
I20250109 20:58:07.834107 17466 tablet_server.cc:178] TabletServer@127.17.14.130:0 shutting down...
I20250109 20:58:07.834715 18551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bd2b37a3e154ffea27c42bd78441e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:07.837366 18536 ts_tablet_manager.cc:1428] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20250109 20:58:07.839068 18295 tablet_copy_service.cc:342] P c33d4d09803d48f99f5f4bf8ac33d402: Request end of tablet copy session f5b05b0a86c1457f8a1d0debf65cecb4-a9942465bba148559c8310fecb210d09 received from {username='slave'} at 127.0.0.1:34358
I20250109 20:58:07.839533 18295 tablet_copy_service.cc:434] P c33d4d09803d48f99f5f4bf8ac33d402: ending tablet copy session f5b05b0a86c1457f8a1d0debf65cecb4-a9942465bba148559c8310fecb210d09 on tablet a9942465bba148559c8310fecb210d09 with peer f5b05b0a86c1457f8a1d0debf65cecb4
I20250109 20:58:07.853296 18551 tablet_copy_client.cc:806] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Starting download of 0 data blocks...
I20250109 20:58:07.853924 18551 tablet_copy_client.cc:670] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Starting download of 1 WAL segments...
I20250109 20:58:07.857869 18551 tablet_copy_client.cc:538] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 20:58:07.882715 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:58:07.883468 17466 tablet_replica.cc:331] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe: stopping tablet replica
I20250109 20:58:07.884030 17466 raft_consensus.cc:2238] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:58:07.884919 17466 raft_consensus.cc:2267] T 19f8261bc37f4e8ea89f4c02dc3c4478 P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.886873 17466 tablet_replica.cc:331] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe: stopping tablet replica
I20250109 20:58:07.887745 17466 raft_consensus.cc:2238] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.888258 17466 raft_consensus.cc:2267] T 7500274b7719441596814acc45fa3028 P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.890291 17466 tablet_replica.cc:331] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe: stopping tablet replica
I20250109 20:58:07.890997 17466 raft_consensus.cc:2238] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.891619 17466 raft_consensus.cc:2267] T 7bd2b37a3e154ffea27c42bd78441e4a P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.892127 18551 tablet_bootstrap.cc:492] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4: Bootstrap starting.
I20250109 20:58:07.894480 17466 tablet_replica.cc:331] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe: stopping tablet replica
I20250109 20:58:07.895285 17466 raft_consensus.cc:2238] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.895838 17466 raft_consensus.cc:2267] T a9942465bba148559c8310fecb210d09 P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.898368 17466 tablet_replica.cc:331] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe: stopping tablet replica
I20250109 20:58:07.899145 17466 raft_consensus.cc:2238] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.899613 17466 raft_consensus.cc:2267] T bec5186b23f94af985e2ede96c30b0dc P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.901878 17466 tablet_replica.cc:331] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe: stopping tablet replica
I20250109 20:58:07.902374 17466 raft_consensus.cc:2238] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:58:07.903091 17466 raft_consensus.cc:2267] T 1e6f9e2410d440079d9ff8e622c7369a P 45c1b06aef574962914c99b98dcbd5fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.905071 17466 tablet_replica.cc:331] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe: stopping tablet replica
I20250109 20:58:07.905764 17466 raft_consensus.cc:2238] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 2 LEADER]: Raft consensus shutting down.
I20250109 20:58:07.906628 17466 raft_consensus.cc:2267] T 40bb82b56471437eba54b55ebb6c20ce P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.909075 17466 tablet_replica.cc:331] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe: stopping tablet replica
I20250109 20:58:07.909679 17466 raft_consensus.cc:2238] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.910138 17466 raft_consensus.cc:2267] T 762caa2bc7564c9d88448466f319a2bf P 45c1b06aef574962914c99b98dcbd5fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.911123 18551 tablet_bootstrap.cc:492] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:58:07.912003 18551 tablet_bootstrap.cc:492] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4: Bootstrap complete.
I20250109 20:58:07.912644 18551 ts_tablet_manager.cc:1397] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.008s
I20250109 20:58:07.915334 18551 raft_consensus.cc:357] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } }
I20250109 20:58:07.916188 18551 raft_consensus.cc:738] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5b05b0a86c1457f8a1d0debf65cecb4, State: Initialized, Role: LEARNER
I20250109 20:58:07.916827 18551 consensus_queue.cc:260] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c84ca066a3a647558af06e1ddf72cfcb" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44553 } attrs { replace: true } } peers { permanent_uuid: "45c1b06aef574962914c99b98dcbd5fe" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 36803 } } peers { permanent_uuid: "c33d4d09803d48f99f5f4bf8ac33d402" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33533 } } peers { permanent_uuid: "f5b05b0a86c1457f8a1d0debf65cecb4" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 41789 } attrs { promote: true } }
I20250109 20:58:07.920804 18551 ts_tablet_manager.cc:1428] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250109 20:58:07.922441 18295 tablet_copy_service.cc:342] P c33d4d09803d48f99f5f4bf8ac33d402: Request end of tablet copy session f5b05b0a86c1457f8a1d0debf65cecb4-7bd2b37a3e154ffea27c42bd78441e4a received from {username='slave'} at 127.0.0.1:34358
I20250109 20:58:07.922816 18295 tablet_copy_service.cc:434] P c33d4d09803d48f99f5f4bf8ac33d402: ending tablet copy session f5b05b0a86c1457f8a1d0debf65cecb4-7bd2b37a3e154ffea27c42bd78441e4a on tablet 7bd2b37a3e154ffea27c42bd78441e4a with peer f5b05b0a86c1457f8a1d0debf65cecb4
I20250109 20:58:07.942886 17466 tablet_server.cc:195] TabletServer@127.17.14.130:0 shutdown complete.
I20250109 20:58:07.962901 17466 tablet_server.cc:178] TabletServer@127.17.14.131:0 shutting down...
W20250109 20:58:07.968839 18082 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250109 20:58:07.969271 18082 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250109 20:58:07.976825 18082 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250109 20:58:07.977155 18082 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250109 20:58:07.986383 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:58:07.987036 17466 tablet_replica.cc:331] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402: stopping tablet replica
I20250109 20:58:07.987565 17466 raft_consensus.cc:2238] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.987974 17466 raft_consensus.cc:2267] T 19f8261bc37f4e8ea89f4c02dc3c4478 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.989637 17466 tablet_replica.cc:331] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402: stopping tablet replica
I20250109 20:58:07.990123 17466 raft_consensus.cc:2238] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.990525 17466 raft_consensus.cc:2267] T bec5186b23f94af985e2ede96c30b0dc P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.992316 17466 tablet_replica.cc:331] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402: stopping tablet replica
I20250109 20:58:07.992802 17466 raft_consensus.cc:2238] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:07.993217 17466 raft_consensus.cc:2267] T 7500274b7719441596814acc45fa3028 P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.995033 17466 tablet_replica.cc:331] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402: stopping tablet replica
I20250109 20:58:07.995505 17466 raft_consensus.cc:2238] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:58:07.996323 17466 raft_consensus.cc:2267] T a9942465bba148559c8310fecb210d09 P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:07.998195 17466 tablet_replica.cc:331] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402: stopping tablet replica
I20250109 20:58:07.998662 17466 raft_consensus.cc:2238] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:58:07.999462 17466 raft_consensus.cc:2267] T 7bd2b37a3e154ffea27c42bd78441e4a P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:08.001197 17466 tablet_replica.cc:331] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402: stopping tablet replica
I20250109 20:58:08.001708 17466 raft_consensus.cc:2238] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:08.002118 17466 raft_consensus.cc:2267] T 762caa2bc7564c9d88448466f319a2bf P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:08.003576 17466 tablet_replica.cc:331] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402: stopping tablet replica
I20250109 20:58:08.003991 17466 raft_consensus.cc:2238] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:08.004338 17466 raft_consensus.cc:2267] T 40bb82b56471437eba54b55ebb6c20ce P c33d4d09803d48f99f5f4bf8ac33d402 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:08.005936 17466 tablet_replica.cc:331] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402: stopping tablet replica
I20250109 20:58:08.006340 17466 raft_consensus.cc:2238] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:08.006675 17466 raft_consensus.cc:2267] T 1e6f9e2410d440079d9ff8e622c7369a P c33d4d09803d48f99f5f4bf8ac33d402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:08.031494 17466 tablet_server.cc:195] TabletServer@127.17.14.131:0 shutdown complete.
I20250109 20:58:08.047515 17466 tablet_server.cc:178] TabletServer@127.17.14.132:0 shutting down...
I20250109 20:58:08.063203 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:58:08.064190 17466 tablet_replica.cc:331] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4: stopping tablet replica
I20250109 20:58:08.064793 17466 raft_consensus.cc:2238] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4 [term 1 LEARNER]: Raft consensus shutting down.
I20250109 20:58:08.065155 17466 raft_consensus.cc:2267] T 7bd2b37a3e154ffea27c42bd78441e4a P f5b05b0a86c1457f8a1d0debf65cecb4 [term 1 LEARNER]: Raft consensus is shut down!
I20250109 20:58:08.066922 17466 tablet_replica.cc:331] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4: stopping tablet replica
I20250109 20:58:08.067386 17466 raft_consensus.cc:2238] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4 [term 1 LEARNER]: Raft consensus shutting down.
I20250109 20:58:08.067687 17466 raft_consensus.cc:2267] T a9942465bba148559c8310fecb210d09 P f5b05b0a86c1457f8a1d0debf65cecb4 [term 1 LEARNER]: Raft consensus is shut down!
I20250109 20:58:08.069269 17466 tablet_replica.cc:331] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4: stopping tablet replica
I20250109 20:58:08.069706 17466 raft_consensus.cc:2238] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4 [term 2 LEARNER]: Raft consensus shutting down.
I20250109 20:58:08.070006 17466 raft_consensus.cc:2267] T bec5186b23f94af985e2ede96c30b0dc P f5b05b0a86c1457f8a1d0debf65cecb4 [term 2 LEARNER]: Raft consensus is shut down!
I20250109 20:58:08.089674 17466 tablet_server.cc:195] TabletServer@127.17.14.132:0 shutdown complete.
I20250109 20:58:08.099853 17466 master.cc:537] Master@127.17.14.190:45483 shutting down...
I20250109 20:58:08.115964 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:58:08.116456 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:08.116752 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6d150a348e194a0abce748c10c8fb2b4: stopping tablet replica
I20250109 20:58:08.135869 17466 master.cc:559] Master@127.17.14.190:45483 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (9378 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20250109 20:58:08.173646 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:37891,127.17.14.189:43243,127.17.14.188:34437
I20250109 20:58:08.175498 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:08.180799 18554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:08.181463 18555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:08.182827 18557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:08.183441 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:08.184212 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:08.184381 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:08.184525 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456288184507 us; error 0 us; skew 500 ppm
I20250109 20:58:08.185002 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:08.187330 17466 webserver.cc:458] Webserver started at http://127.17.14.190:38147/ using document root <none> and password file <none>
I20250109 20:58:08.187786 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:08.187963 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:08.188206 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:08.189213 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "01b462cdf3d0443a84535c8dec8c6c9a"
format_stamp: "Formatted at 2025-01-09 20:58:08 on dist-test-slave-j54z"
I20250109 20:58:08.193325 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250109 20:58:08.196633 18562 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:08.197463 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:58:08.197839 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "01b462cdf3d0443a84535c8dec8c6c9a"
format_stamp: "Formatted at 2025-01-09 20:58:08 on dist-test-slave-j54z"
I20250109 20:58:08.198226 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:08.211722 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:08.212788 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:08.248596 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:37891
I20250109 20:58:08.248683 18613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:37891 every 8 connection(s)
I20250109 20:58:08.252241 18614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:08.252421 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:08.257951 18616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:08.258723 18614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:58:08.259311 18617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:08.261291 18619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:08.262590 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:08.263345 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:08.263613 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:08.263811 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456288263795 us; error 0 us; skew 500 ppm
I20250109 20:58:08.264422 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:08.267181 17466 webserver.cc:458] Webserver started at http://127.17.14.189:42025/ using document root <none> and password file <none>
I20250109 20:58:08.267760 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:08.267930 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:08.268186 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:08.269232 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-1-root/instance:
uuid: "964362e3bdcc441c8315934ab5e26c98"
format_stamp: "Formatted at 2025-01-09 20:58:08 on dist-test-slave-j54z"
I20250109 20:58:08.270545 18614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:58:08.274335 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
W20250109 20:58:08.275806 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.14.189:43243: Network error: Client connection negotiation failed: client connection to 127.17.14.189:43243: connect: Connection refused (error 111)
I20250109 20:58:08.277738 18627 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:08.278447 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:58:08.278695 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-1-root
uuid: "964362e3bdcc441c8315934ab5e26c98"
format_stamp: "Formatted at 2025-01-09 20:58:08 on dist-test-slave-j54z"
I20250109 20:58:08.278957 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:08.303462 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:08.304570 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:08.316337 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } attempt: 1
W20250109 20:58:08.320504 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.14.189:43243: Network error: Client connection negotiation failed: client connection to 127.17.14.189:43243: connect: Connection refused (error 111)
I20250109 20:58:08.339701 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.189:43243
I20250109 20:58:08.339794 18678 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.189:43243 every 8 connection(s)
I20250109 20:58:08.343186 18679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:08.343449 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:08.349149 18681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:08.349790 18679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:58:08.350504 18682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:08.352319 17466 server_base.cc:1034] running on GCE node
W20250109 20:58:08.353972 18684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:08.354835 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:08.355101 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:08.355312 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456288355291 us; error 0 us; skew 500 ppm
I20250109 20:58:08.355885 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:08.359164 17466 webserver.cc:458] Webserver started at http://127.17.14.188:46415/ using document root <none> and password file <none>
I20250109 20:58:08.359802 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:08.360003 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:08.360261 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:08.361650 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-2-root/instance:
uuid: "682f1f1f507c4b7882d9605ebf01b975"
format_stamp: "Formatted at 2025-01-09 20:58:08 on dist-test-slave-j54z"
I20250109 20:58:08.362324 18679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:58:08.366657 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20250109 20:58:08.370183 18691 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:08.370975 17466 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250109 20:58:08.371304 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-2-root
uuid: "682f1f1f507c4b7882d9605ebf01b975"
format_stamp: "Formatted at 2025-01-09 20:58:08 on dist-test-slave-j54z"
I20250109 20:58:08.371623 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:08.377930 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } attempt: 2
I20250109 20:58:08.380209 18679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:58:08.386535 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20250109 20:58:08.386507 18679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.14.188:34437: Network error: Client connection negotiation failed: client connection to 127.17.14.188:34437: connect: Connection refused (error 111)
I20250109 20:58:08.388041 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:08.388724 18614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:58:08.392911 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.14.188:34437: Network error: Client connection negotiation failed: client connection to 127.17.14.188:34437: connect: Connection refused (error 111)
I20250109 20:58:08.412283 18679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } attempt: 1
I20250109 20:58:08.416335 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } attempt: 1
W20250109 20:58:08.417119 18679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.14.188:34437: Network error: Client connection negotiation failed: client connection to 127.17.14.188:34437: connect: Connection refused (error 111)
W20250109 20:58:08.420921 18614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.14.188:34437: Network error: Client connection negotiation failed: client connection to 127.17.14.188:34437: connect: Connection refused (error 111)
I20250109 20:58:08.427778 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.188:34437
I20250109 20:58:08.427886 18743 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.188:34437 every 8 connection(s)
I20250109 20:58:08.430677 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 20:58:08.431335 18744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:08.436790 18744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:58:08.446911 18744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:58:08.454692 18679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } attempt: 2
I20250109 20:58:08.456015 18744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:58:08.470588 18679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98: Bootstrap starting.
I20250109 20:58:08.472066 18744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975: Bootstrap starting.
I20250109 20:58:08.474936 18679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:08.476441 18744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:08.479126 18679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98: No bootstrap required, opened a new log
I20250109 20:58:08.480862 18744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975: No bootstrap required, opened a new log
I20250109 20:58:08.481209 18679 raft_consensus.cc:357] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:08.481802 18679 raft_consensus.cc:383] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:08.482019 18679 raft_consensus.cc:738] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964362e3bdcc441c8315934ab5e26c98, State: Initialized, Role: FOLLOWER
I20250109 20:58:08.482529 18679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:08.482843 18744 raft_consensus.cc:357] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:08.483407 18744 raft_consensus.cc:383] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:08.483920 18744 raft_consensus.cc:738] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682f1f1f507c4b7882d9605ebf01b975, State: Initialized, Role: FOLLOWER
I20250109 20:58:08.484504 18744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:08.484331 18751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:08.485203 18751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:58:08.485926 18679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:58:08.486377 18752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:08.487203 18752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:58:08.487908 18744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:58:08.502558 18614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } attempt: 2
W20250109 20:58:08.503327 18769 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 20:58:08.503598 18769 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250109 20:58:08.505270 18774 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 20:58:08.505630 18774 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 20:58:08.517645 18614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a: Bootstrap starting.
I20250109 20:58:08.521631 18614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:08.525254 18614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a: No bootstrap required, opened a new log
I20250109 20:58:08.527175 18614 raft_consensus.cc:357] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:08.527693 18614 raft_consensus.cc:383] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:08.527930 18614 raft_consensus.cc:738] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b462cdf3d0443a84535c8dec8c6c9a, State: Initialized, Role: FOLLOWER
I20250109 20:58:08.528422 18614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:08.529979 18776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:08.530539 18776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:58:08.531137 18614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:58:08.540633 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250109 20:58:08.540841 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250109 20:58:08.542186 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:08.543570 18787 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 20:58:08.543848 18787 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250109 20:58:08.548303 18788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:08.549914 18789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:08.551132 18791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:08.551997 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:08.552721 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:08.552898 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:08.553023 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456288553012 us; error 0 us; skew 500 ppm
I20250109 20:58:08.553433 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:08.555738 17466 webserver.cc:458] Webserver started at http://127.17.14.129:43161/ using document root <none> and password file <none>
I20250109 20:58:08.556164 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:08.556331 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:08.556542 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:08.557686 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-0-root/instance:
uuid: "209712ca30bf49a58d4df009d099752b"
format_stamp: "Formatted at 2025-01-09 20:58:08 on dist-test-slave-j54z"
I20250109 20:58:08.561913 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250109 20:58:08.564806 18796 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:08.565590 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 20:58:08.565860 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "209712ca30bf49a58d4df009d099752b"
format_stamp: "Formatted at 2025-01-09 20:58:08 on dist-test-slave-j54z"
I20250109 20:58:08.566136 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:08.611084 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:08.612202 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:08.614439 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:58:08.616705 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:58:08.616983 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:08.617272 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:58:08.617471 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:08.654865 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:33871
I20250109 20:58:08.654948 18858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:33871 every 8 connection(s)
I20250109 20:58:08.678123 18860 heartbeater.cc:346] Connected to a master server at 127.17.14.189:43243
I20250109 20:58:08.678534 18860 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:08.679231 18860 heartbeater.cc:510] Master 127.17.14.189:43243 requested a full tablet report, sending...
I20250109 20:58:08.680716 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 20:58:08.681296 18859 heartbeater.cc:346] Connected to a master server at 127.17.14.188:34437
I20250109 20:58:08.681751 18859 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:08.682091 18644 ts_manager.cc:194] Registered new tserver with Master: 209712ca30bf49a58d4df009d099752b (127.17.14.129:33871)
I20250109 20:58:08.682554 18859 heartbeater.cc:510] Master 127.17.14.188:34437 requested a full tablet report, sending...
I20250109 20:58:08.685146 18862 heartbeater.cc:346] Connected to a master server at 127.17.14.190:37891
I20250109 20:58:08.685235 18709 ts_manager.cc:194] Registered new tserver with Master: 209712ca30bf49a58d4df009d099752b (127.17.14.129:33871)
I20250109 20:58:08.685616 18862 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:08.686472 18862 heartbeater.cc:510] Master 127.17.14.190:37891 requested a full tablet report, sending...
I20250109 20:58:08.689296 18579 ts_manager.cc:194] Registered new tserver with Master: 209712ca30bf49a58d4df009d099752b (127.17.14.129:33871)
W20250109 20:58:08.690639 18866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:08.692343 18869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:08.693009 17466 server_base.cc:1034] running on GCE node
W20250109 20:58:08.694303 18867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:08.695143 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:08.695338 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:08.695530 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456288695511 us; error 0 us; skew 500 ppm
I20250109 20:58:08.696130 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:08.698478 17466 webserver.cc:458] Webserver started at http://127.17.14.130:44305/ using document root <none> and password file <none>
I20250109 20:58:08.698940 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:08.699095 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:08.699292 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:08.700275 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-1-root/instance:
uuid: "279644ed6984453b860ccd5bb5c4addc"
format_stamp: "Formatted at 2025-01-09 20:58:08 on dist-test-slave-j54z"
I20250109 20:58:08.704558 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 20:58:08.707571 18874 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:08.708267 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 20:58:08.708504 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "279644ed6984453b860ccd5bb5c4addc"
format_stamp: "Formatted at 2025-01-09 20:58:08 on dist-test-slave-j54z"
I20250109 20:58:08.708743 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:08.729465 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:08.730645 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:08.731985 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:58:08.734261 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:58:08.734473 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:08.734719 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:58:08.734886 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:08.760730 18776 raft_consensus.cc:491] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:08.761209 18776 raft_consensus.cc:513] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:08.763037 18776 leader_election.cc:290] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 964362e3bdcc441c8315934ab5e26c98 (127.17.14.189:43243), 682f1f1f507c4b7882d9605ebf01b975 (127.17.14.188:34437)
I20250109 20:58:08.763909 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964362e3bdcc441c8315934ab5e26c98" is_pre_election: true
I20250109 20:58:08.764567 18654 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01b462cdf3d0443a84535c8dec8c6c9a in term 0.
I20250109 20:58:08.764312 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682f1f1f507c4b7882d9605ebf01b975" is_pre_election: true
I20250109 20:58:08.765260 18719 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01b462cdf3d0443a84535c8dec8c6c9a in term 0.
I20250109 20:58:08.765784 18564 leader_election.cc:304] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b462cdf3d0443a84535c8dec8c6c9a, 964362e3bdcc441c8315934ab5e26c98; no voters:
I20250109 20:58:08.766522 18776 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:58:08.766875 18776 raft_consensus.cc:491] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:58:08.767216 18776 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:08.777454 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:33905
I20250109 20:58:08.777658 18936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:33905 every 8 connection(s)
I20250109 20:58:08.789562 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 20:58:08.801695 18937 heartbeater.cc:346] Connected to a master server at 127.17.14.188:34437
I20250109 20:58:08.802234 18937 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:08.803018 18939 heartbeater.cc:346] Connected to a master server at 127.17.14.190:37891
I20250109 20:58:08.803431 18939 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:08.803323 18937 heartbeater.cc:510] Master 127.17.14.188:34437 requested a full tablet report, sending...
I20250109 20:58:08.804415 18939 heartbeater.cc:510] Master 127.17.14.190:37891 requested a full tablet report, sending...
I20250109 20:58:08.807168 18709 ts_manager.cc:194] Registered new tserver with Master: 279644ed6984453b860ccd5bb5c4addc (127.17.14.130:33905)
I20250109 20:58:08.809346 18938 heartbeater.cc:346] Connected to a master server at 127.17.14.189:43243
I20250109 20:58:08.809859 18938 heartbeater.cc:463] Registering TS with master...
W20250109 20:58:08.810290 18944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:08.811100 18938 heartbeater.cc:510] Master 127.17.14.189:43243 requested a full tablet report, sending...
W20250109 20:58:08.812319 18945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:08.814028 18644 ts_manager.cc:194] Registered new tserver with Master: 279644ed6984453b860ccd5bb5c4addc (127.17.14.130:33905)
W20250109 20:58:08.814626 18947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:08.815719 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:08.816592 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:08.816933 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:08.816515 18776 raft_consensus.cc:513] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:08.817150 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456288817133 us; error 0 us; skew 500 ppm
I20250109 20:58:08.817911 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:08.818708 18579 ts_manager.cc:194] Registered new tserver with Master: 279644ed6984453b860ccd5bb5c4addc (127.17.14.130:33905)
I20250109 20:58:08.819690 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682f1f1f507c4b7882d9605ebf01b975"
I20250109 20:58:08.818589 18776 leader_election.cc:290] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [CANDIDATE]: Term 1 election: Requested vote from peers 964362e3bdcc441c8315934ab5e26c98 (127.17.14.189:43243), 682f1f1f507c4b7882d9605ebf01b975 (127.17.14.188:34437)
I20250109 20:58:08.820446 18719 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:08.820533 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964362e3bdcc441c8315934ab5e26c98"
I20250109 20:58:08.821197 18654 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:08.822767 17466 webserver.cc:458] Webserver started at http://127.17.14.131:45181/ using document root <none> and password file <none>
I20250109 20:58:08.823421 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:08.823665 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:08.823967 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:08.825315 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-2-root/instance:
uuid: "a1863c1c9da7421b83fd857841fc8142"
format_stamp: "Formatted at 2025-01-09 20:58:08 on dist-test-slave-j54z"
I20250109 20:58:08.825918 18654 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b462cdf3d0443a84535c8dec8c6c9a in term 1.
I20250109 20:58:08.826784 18564 leader_election.cc:304] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b462cdf3d0443a84535c8dec8c6c9a, 964362e3bdcc441c8315934ab5e26c98; no voters:
I20250109 20:58:08.827417 18776 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:08.827538 18719 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b462cdf3d0443a84535c8dec8c6c9a in term 1.
I20250109 20:58:08.828583 18776 raft_consensus.cc:695] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 1 LEADER]: Becoming Leader. State: Replica: 01b462cdf3d0443a84535c8dec8c6c9a, State: Running, Role: LEADER
I20250109 20:58:08.829316 18776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:08.831051 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250109 20:58:08.833108 18951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01b462cdf3d0443a84535c8dec8c6c9a. Latest consensus state: current_term: 1 leader_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:08.833758 18951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: This master's current role is: LEADER
I20250109 20:58:08.835393 18954 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:58:08.835669 18955 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:08.836568 17466 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250109 20:58:08.836855 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "a1863c1c9da7421b83fd857841fc8142"
format_stamp: "Formatted at 2025-01-09 20:58:08 on dist-test-slave-j54z"
I20250109 20:58:08.837195 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:08.840270 18954 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:58:08.848275 18654 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 1 FOLLOWER]: Refusing update from remote peer 01b462cdf3d0443a84535c8dec8c6c9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:58:08.848495 18719 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 1 FOLLOWER]: Refusing update from remote peer 01b462cdf3d0443a84535c8dec8c6c9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:58:08.849697 18776 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:08.850347 18951 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:08.859012 18751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01b462cdf3d0443a84535c8dec8c6c9a. Latest consensus state: current_term: 1 leader_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:08.859916 18751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:58:08.861852 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:08.863624 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:08.863837 18776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:08.864602 18776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: This master's current role is: LEADER
I20250109 20:58:08.864662 18954 catalog_manager.cc:1348] Generated new cluster ID: 2cb8ba59dc0b43a5b20b8e6eab387656
I20250109 20:58:08.865043 18954 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:58:08.871153 18752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01b462cdf3d0443a84535c8dec8c6c9a. Latest consensus state: current_term: 1 leader_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:08.872195 18752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:58:08.875484 18951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:08.876523 18951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: This master's current role is: LEADER
I20250109 20:58:08.881141 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:58:08.888489 18751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:08.889293 18751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:58:08.895473 18752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:08.896219 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:58:08.896401 18752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:58:08.896555 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:08.896889 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:58:08.897140 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.001s
I20250109 20:58:08.906213 18954 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 20:58:08.907902 18954 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:58:08.932488 18954 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a: Generated new TSK 0
I20250109 20:58:08.933209 18954 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:58:08.962898 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:37903
I20250109 20:58:08.962987 19026 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:37903 every 8 connection(s)
I20250109 20:58:08.981263 19029 heartbeater.cc:346] Connected to a master server at 127.17.14.190:37891
I20250109 20:58:08.981452 19028 heartbeater.cc:346] Connected to a master server at 127.17.14.189:43243
I20250109 20:58:08.981722 19029 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:08.981951 19028 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:08.982666 19029 heartbeater.cc:510] Master 127.17.14.190:37891 requested a full tablet report, sending...
I20250109 20:58:08.982749 19028 heartbeater.cc:510] Master 127.17.14.189:43243 requested a full tablet report, sending...
I20250109 20:58:08.984997 18579 ts_manager.cc:194] Registered new tserver with Master: a1863c1c9da7421b83fd857841fc8142 (127.17.14.131:37903)
I20250109 20:58:08.985119 19027 heartbeater.cc:346] Connected to a master server at 127.17.14.188:34437
I20250109 20:58:08.985342 18644 ts_manager.cc:194] Registered new tserver with Master: a1863c1c9da7421b83fd857841fc8142 (127.17.14.131:37903)
I20250109 20:58:08.985615 19027 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:08.986606 19027 heartbeater.cc:510] Master 127.17.14.188:34437 requested a full tablet report, sending...
I20250109 20:58:08.987412 18579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48620
I20250109 20:58:08.988813 18709 ts_manager.cc:194] Registered new tserver with Master: a1863c1c9da7421b83fd857841fc8142 (127.17.14.131:37903)
I20250109 20:58:08.989300 17466 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.02085529s
I20250109 20:58:10.707547 18751 raft_consensus.cc:491] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01b462cdf3d0443a84535c8dec8c6c9a)
I20250109 20:58:10.708289 18751 raft_consensus.cc:513] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:10.710922 18751 leader_election.cc:290] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01b462cdf3d0443a84535c8dec8c6c9a (127.17.14.190:37891), 682f1f1f507c4b7882d9605ebf01b975 (127.17.14.188:34437)
I20250109 20:58:10.738049 18774 catalog_manager.cc:1260] Loaded cluster ID: 2cb8ba59dc0b43a5b20b8e6eab387656
I20250109 20:58:10.738600 18774 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975: loading cluster ID for follower catalog manager: success
I20250109 20:58:10.741180 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "964362e3bdcc441c8315934ab5e26c98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "682f1f1f507c4b7882d9605ebf01b975" is_pre_election: true
I20250109 20:58:10.741881 18719 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 964362e3bdcc441c8315934ab5e26c98 in term 1.
I20250109 20:58:10.742847 18629 leader_election.cc:304] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682f1f1f507c4b7882d9605ebf01b975, 964362e3bdcc441c8315934ab5e26c98; no voters:
I20250109 20:58:10.743570 18751 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250109 20:58:10.743877 18751 raft_consensus.cc:491] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 01b462cdf3d0443a84535c8dec8c6c9a)
I20250109 20:58:10.744148 18751 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:10.747067 18774 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975: acquiring CA information for follower catalog manager: success
I20250109 20:58:10.748405 18786 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:10.750317 18751 raft_consensus.cc:513] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:10.754554 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "964362e3bdcc441c8315934ab5e26c98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "682f1f1f507c4b7882d9605ebf01b975"
I20250109 20:58:10.755033 18719 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:10.753939 18751 leader_election.cc:290] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [CANDIDATE]: Term 2 election: Requested vote from peers 01b462cdf3d0443a84535c8dec8c6c9a (127.17.14.190:37891), 682f1f1f507c4b7882d9605ebf01b975 (127.17.14.188:34437)
I20250109 20:58:10.758483 19029 heartbeater.cc:502] Master 127.17.14.190:37891 was elected leader, sending a full tablet report...
I20250109 20:58:10.758548 18774 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:58:10.765204 18719 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 964362e3bdcc441c8315934ab5e26c98 in term 2.
I20250109 20:58:10.808180 18629 leader_election.cc:304] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682f1f1f507c4b7882d9605ebf01b975, 964362e3bdcc441c8315934ab5e26c98; no voters:
I20250109 20:58:10.813035 18579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48610
I20250109 20:58:10.818548 18769 catalog_manager.cc:1260] Loaded cluster ID: 2cb8ba59dc0b43a5b20b8e6eab387656
I20250109 20:58:10.819015 18769 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98: loading cluster ID for follower catalog manager: success
I20250109 20:58:10.819380 18751 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 20:58:10.823837 18752 raft_consensus.cc:491] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:10.824757 18752 raft_consensus.cc:513] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:10.829161 18752 leader_election.cc:290] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 01b462cdf3d0443a84535c8dec8c6c9a (127.17.14.190:37891), 964362e3bdcc441c8315934ab5e26c98 (127.17.14.189:43243)
W20250109 20:58:10.835053 18933 debug-util.cc:398] Leaking SignalData structure 0x7b0800216da0 after lost signal to thread 17469
I20250109 20:58:10.834856 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "682f1f1f507c4b7882d9605ebf01b975" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "964362e3bdcc441c8315934ab5e26c98" is_pre_election: true
W20250109 20:58:10.836941 18933 debug-util.cc:398] Leaking SignalData structure 0x7b0800044140 after lost signal to thread 18613
I20250109 20:58:10.837929 18769 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98: acquiring CA information for follower catalog manager: success
W20250109 20:58:10.838627 18933 debug-util.cc:398] Leaking SignalData structure 0x7b08001d0c00 after lost signal to thread 18678
W20250109 20:58:10.840802 18933 debug-util.cc:398] Leaking SignalData structure 0x7b08000e8960 after lost signal to thread 18743
I20250109 20:58:10.841990 18751 raft_consensus.cc:695] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 2 LEADER]: Becoming Leader. State: Replica: 964362e3bdcc441c8315934ab5e26c98, State: Running, Role: LEADER
I20250109 20:58:10.843245 18751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
W20250109 20:58:10.843986 18933 debug-util.cc:398] Leaking SignalData structure 0x7b08001abb20 after lost signal to thread 18858
I20250109 20:58:10.845515 18769 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250109 20:58:10.846596 18933 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba8e0 after lost signal to thread 18936
W20250109 20:58:10.848996 18933 debug-util.cc:398] Leaking SignalData structure 0x7b0800200760 after lost signal to thread 19026
I20250109 20:58:10.849493 18579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48618
I20250109 20:58:10.850610 18719 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 2 FOLLOWER]: Rejecting Update request from peer 01b462cdf3d0443a84535c8dec8c6c9a for earlier term 1. Current term is 2. Ops: []
I20250109 20:58:10.855844 18589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "682f1f1f507c4b7882d9605ebf01b975" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" is_pre_election: true
I20250109 20:58:10.858214 18587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "964362e3bdcc441c8315934ab5e26c98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01b462cdf3d0443a84535c8dec8c6c9a"
I20250109 20:58:10.859977 18588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "964362e3bdcc441c8315934ab5e26c98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" is_pre_election: true
I20250109 20:58:10.859622 19038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 964362e3bdcc441c8315934ab5e26c98. Latest consensus state: current_term: 2 leader_uuid: "964362e3bdcc441c8315934ab5e26c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:10.860694 19038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: This master's current role is: LEADER
I20250109 20:58:10.862494 18653 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 2 LEADER]: Rejecting Update request from peer 01b462cdf3d0443a84535c8dec8c6c9a for earlier term 1. Current term is 2. Ops: []
I20250109 20:58:10.862277 19037 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250109 20:58:10.863631 19037 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 1 LEADER]: Stepping down as leader of term 1
I20250109 20:58:10.863899 19040 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:58:10.864007 19037 raft_consensus.cc:738] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 01b462cdf3d0443a84535c8dec8c6c9a, State: Running, Role: LEADER
I20250109 20:58:10.865154 19037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } }
I20250109 20:58:10.866772 19037 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:10.867592 18693 leader_election.cc:304] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 682f1f1f507c4b7882d9605ebf01b975; no voters: 01b462cdf3d0443a84535c8dec8c6c9a, 964362e3bdcc441c8315934ab5e26c98
I20250109 20:58:10.868878 18752 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250109 20:58:10.874106 19040 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:58:10.878948 19040 catalog_manager.cc:1260] Loaded cluster ID: 2cb8ba59dc0b43a5b20b8e6eab387656
I20250109 20:58:10.879238 19040 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:58:10.884560 19040 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:58:10.889320 19040 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98: Loaded TSK: 0
I20250109 20:58:10.890738 19040 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:58:11.378997 18588 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 2 FOLLOWER]: Refusing update from remote peer 964362e3bdcc441c8315934ab5e26c98: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 20:58:11.380332 19041 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:58:11.385499 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 964362e3bdcc441c8315934ab5e26c98. Latest consensus state: current_term: 2 leader_uuid: "964362e3bdcc441c8315934ab5e26c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:11.386519 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:58:11.392133 18719 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 2 FOLLOWER]: Refusing update from remote peer 964362e3bdcc441c8315934ab5e26c98: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 20:58:11.399323 19041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "964362e3bdcc441c8315934ab5e26c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:11.399958 19048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:58:11.400476 19041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: This master's current role is: LEADER
I20250109 20:58:11.403470 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "964362e3bdcc441c8315934ab5e26c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:11.404305 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:58:11.410118 19053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "964362e3bdcc441c8315934ab5e26c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:11.410463 19050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 964362e3bdcc441c8315934ab5e26c98. Latest consensus state: current_term: 2 leader_uuid: "964362e3bdcc441c8315934ab5e26c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:11.411044 19053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:58:11.411110 19050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:58:11.554365 18787 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:58:11.834395 18862 heartbeater.cc:502] Master 127.17.14.190:37891 was elected leader, sending a full tablet report...
I20250109 20:58:11.853255 18939 heartbeater.cc:502] Master 127.17.14.190:37891 was elected leader, sending a full tablet report...
I20250109 20:58:11.889686 19041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "964362e3bdcc441c8315934ab5e26c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b462cdf3d0443a84535c8dec8c6c9a" member_type: VOTER last_known_addr { host: "127.17.14.190" port: 37891 } } peers { permanent_uuid: "964362e3bdcc441c8315934ab5e26c98" member_type: VOTER last_known_addr { host: "127.17.14.189" port: 43243 } } peers { permanent_uuid: "682f1f1f507c4b7882d9605ebf01b975" member_type: VOTER last_known_addr { host: "127.17.14.188" port: 34437 } } }
I20250109 20:58:11.890569 19041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [sys.catalog]: This master's current role is: LEADER
I20250109 20:58:12.733237 18767 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:12.783092 17466 test_util.cc:274] Using random seed: 1015010174
I20250109 20:58:12.827831 18938 heartbeater.cc:502] Master 127.17.14.189:43243 was elected leader, sending a full tablet report...
I20250109 20:58:12.830075 18860 heartbeater.cc:502] Master 127.17.14.189:43243 was elected leader, sending a full tablet report...
I20250109 20:58:12.831571 19028 heartbeater.cc:502] Master 127.17.14.189:43243 was elected leader, sending a full tablet report...
I20250109 20:58:12.842334 18642 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54618:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:58:12.951221 18902 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1107cdf52040a09777af0850712e20 (DEFAULT_TABLE table=test-workload [id=62cceb6cf4bb4c3988f4af37791660f8]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250109 20:58:12.953017 18902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1107cdf52040a09777af0850712e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:12.953372 18992 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0e5f3424b343948fd5009413e6eebc (DEFAULT_TABLE table=test-workload [id=62cceb6cf4bb4c3988f4af37791660f8]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250109 20:58:12.954754 18992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a0e5f3424b343948fd5009413e6eebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:12.961714 18824 tablet_service.cc:1467] Processing CreateTablet for tablet 6bbe427346d5403da058ab3da59a1a85 (DEFAULT_TABLE table=test-workload [id=62cceb6cf4bb4c3988f4af37791660f8]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250109 20:58:12.963239 18824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bbe427346d5403da058ab3da59a1a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:12.980794 19084 tablet_bootstrap.cc:492] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b: Bootstrap starting.
I20250109 20:58:12.983992 19082 tablet_bootstrap.cc:492] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc: Bootstrap starting.
I20250109 20:58:12.990481 19084 tablet_bootstrap.cc:654] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:12.993002 19082 tablet_bootstrap.cc:654] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:12.994901 19083 tablet_bootstrap.cc:492] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142: Bootstrap starting.
I20250109 20:58:13.001855 19083 tablet_bootstrap.cc:654] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:13.001920 19082 tablet_bootstrap.cc:492] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc: No bootstrap required, opened a new log
I20250109 20:58:13.002511 19082 ts_tablet_manager.cc:1397] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20250109 20:58:13.004726 19082 raft_consensus.cc:357] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279644ed6984453b860ccd5bb5c4addc" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 33905 } }
I20250109 20:58:13.005230 19082 raft_consensus.cc:383] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:13.005606 19082 raft_consensus.cc:738] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279644ed6984453b860ccd5bb5c4addc, State: Initialized, Role: FOLLOWER
I20250109 20:58:13.006469 19082 consensus_queue.cc:260] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279644ed6984453b860ccd5bb5c4addc" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 33905 } }
I20250109 20:58:13.007220 19082 raft_consensus.cc:397] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:58:13.007529 19082 raft_consensus.cc:491] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:58:13.007689 19084 tablet_bootstrap.cc:492] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b: No bootstrap required, opened a new log
I20250109 20:58:13.007966 19082 raft_consensus.cc:3054] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:13.008239 19083 tablet_bootstrap.cc:492] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142: No bootstrap required, opened a new log
I20250109 20:58:13.008258 19084 ts_tablet_manager.cc:1397] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.004s
I20250109 20:58:13.009116 19083 ts_tablet_manager.cc:1397] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20250109 20:58:13.011534 19084 raft_consensus.cc:357] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209712ca30bf49a58d4df009d099752b" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 33871 } }
I20250109 20:58:13.012323 19084 raft_consensus.cc:383] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:13.012648 19084 raft_consensus.cc:738] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209712ca30bf49a58d4df009d099752b, State: Initialized, Role: FOLLOWER
I20250109 20:58:13.013302 19084 consensus_queue.cc:260] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209712ca30bf49a58d4df009d099752b" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 33871 } }
I20250109 20:58:13.014027 19084 raft_consensus.cc:397] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:58:13.014339 19084 raft_consensus.cc:491] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:58:13.014518 19083 raft_consensus.cc:357] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1863c1c9da7421b83fd857841fc8142" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 37903 } }
I20250109 20:58:13.014962 19084 raft_consensus.cc:3054] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:13.015141 19083 raft_consensus.cc:383] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:13.015497 19083 raft_consensus.cc:738] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1863c1c9da7421b83fd857841fc8142, State: Initialized, Role: FOLLOWER
I20250109 20:58:13.016016 19083 consensus_queue.cc:260] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1863c1c9da7421b83fd857841fc8142" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 37903 } }
I20250109 20:58:13.016479 19083 raft_consensus.cc:397] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:58:13.016703 19083 raft_consensus.cc:491] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:58:13.016956 19083 raft_consensus.cc:3054] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:13.018700 19082 raft_consensus.cc:513] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279644ed6984453b860ccd5bb5c4addc" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 33905 } }
I20250109 20:58:13.019655 19082 leader_election.cc:304] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 279644ed6984453b860ccd5bb5c4addc; no voters:
I20250109 20:58:13.022414 19084 raft_consensus.cc:513] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209712ca30bf49a58d4df009d099752b" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 33871 } }
I20250109 20:58:13.023176 19084 leader_election.cc:304] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 209712ca30bf49a58d4df009d099752b; no voters:
I20250109 20:58:13.024236 19083 raft_consensus.cc:513] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1863c1c9da7421b83fd857841fc8142" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 37903 } }
I20250109 20:58:13.025147 19083 leader_election.cc:304] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1863c1c9da7421b83fd857841fc8142; no voters:
I20250109 20:58:13.028321 19088 raft_consensus.cc:2798] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:13.028888 19082 leader_election.cc:290] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:58:13.033797 19090 raft_consensus.cc:2798] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:13.034881 19083 leader_election.cc:290] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:58:13.037729 19082 ts_tablet_manager.cc:1428] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc: Time spent starting tablet: real 0.035s user 0.013s sys 0.011s
I20250109 20:58:13.038009 19084 leader_election.cc:290] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:58:13.038479 19089 raft_consensus.cc:2798] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:13.039224 19083 ts_tablet_manager.cc:1428] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142: Time spent starting tablet: real 0.030s user 0.021s sys 0.002s
I20250109 20:58:13.041683 19088 raft_consensus.cc:695] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [term 1 LEADER]: Becoming Leader. State: Replica: 279644ed6984453b860ccd5bb5c4addc, State: Running, Role: LEADER
I20250109 20:58:13.042680 19090 raft_consensus.cc:695] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [term 1 LEADER]: Becoming Leader. State: Replica: a1863c1c9da7421b83fd857841fc8142, State: Running, Role: LEADER
I20250109 20:58:13.042599 19088 consensus_queue.cc:237] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279644ed6984453b860ccd5bb5c4addc" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 33905 } }
I20250109 20:58:13.043408 19090 consensus_queue.cc:237] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1863c1c9da7421b83fd857841fc8142" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 37903 } }
I20250109 20:58:13.048086 19089 raft_consensus.cc:695] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [term 1 LEADER]: Becoming Leader. State: Replica: 209712ca30bf49a58d4df009d099752b, State: Running, Role: LEADER
I20250109 20:58:13.048997 19089 consensus_queue.cc:237] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209712ca30bf49a58d4df009d099752b" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 33871 } }
I20250109 20:58:13.065631 19084 ts_tablet_manager.cc:1428] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b: Time spent starting tablet: real 0.057s user 0.038s sys 0.016s
I20250109 20:58:13.073001 18642 catalog_manager.cc:5526] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1863c1c9da7421b83fd857841fc8142 (127.17.14.131). New cstate: current_term: 1 leader_uuid: "a1863c1c9da7421b83fd857841fc8142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1863c1c9da7421b83fd857841fc8142" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 37903 } health_report { overall_health: HEALTHY } } }
I20250109 20:58:13.075564 18643 catalog_manager.cc:5526] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b reported cstate change: term changed from 0 to 1, leader changed from <none> to 209712ca30bf49a58d4df009d099752b (127.17.14.129). New cstate: current_term: 1 leader_uuid: "209712ca30bf49a58d4df009d099752b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209712ca30bf49a58d4df009d099752b" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 33871 } health_report { overall_health: HEALTHY } } }
I20250109 20:58:13.122304 18644 catalog_manager.cc:5526] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc reported cstate change: term changed from 0 to 1, leader changed from <none> to 279644ed6984453b860ccd5bb5c4addc (127.17.14.130). New cstate: current_term: 1 leader_uuid: "279644ed6984453b860ccd5bb5c4addc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279644ed6984453b860ccd5bb5c4addc" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 33905 } health_report { overall_health: HEALTHY } } }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:373: Failure
Expected equality of these values:
0
NumLoopIterations(i)
Which is: 1
I20250109 20:58:13.245239 17466 tablet_server.cc:178] TabletServer@127.17.14.129:0 shutting down...
I20250109 20:58:13.283243 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:58:13.284193 17466 tablet_replica.cc:331] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b: stopping tablet replica
I20250109 20:58:13.285102 17466 raft_consensus.cc:2238] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:58:13.285806 17466 raft_consensus.cc:2267] T 6bbe427346d5403da058ab3da59a1a85 P 209712ca30bf49a58d4df009d099752b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:13.297444 17466 tablet_server.cc:195] TabletServer@127.17.14.129:0 shutdown complete.
I20250109 20:58:13.309888 17466 tablet_server.cc:178] TabletServer@127.17.14.130:0 shutting down...
I20250109 20:58:13.339426 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:58:13.340222 17466 tablet_replica.cc:331] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc: stopping tablet replica
I20250109 20:58:13.340971 17466 raft_consensus.cc:2238] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:58:13.341377 17466 raft_consensus.cc:2267] T 3f1107cdf52040a09777af0850712e20 P 279644ed6984453b860ccd5bb5c4addc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:13.866657 17466 tablet_server.cc:195] TabletServer@127.17.14.130:0 shutdown complete.
I20250109 20:58:13.875162 17466 tablet_server.cc:178] TabletServer@127.17.14.131:0 shutting down...
I20250109 20:58:13.893674 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:58:13.894284 17466 tablet_replica.cc:331] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142: stopping tablet replica
I20250109 20:58:13.894827 17466 raft_consensus.cc:2238] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:58:13.895196 17466 raft_consensus.cc:2267] T 4a0e5f3424b343948fd5009413e6eebc P a1863c1c9da7421b83fd857841fc8142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:13.912830 17466 tablet_server.cc:195] TabletServer@127.17.14.131:0 shutdown complete.
I20250109 20:58:13.921164 17466 master.cc:537] Master@127.17.14.190:37891 shutting down...
I20250109 20:58:13.938192 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:13.938686 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:13.938982 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P 01b462cdf3d0443a84535c8dec8c6c9a: stopping tablet replica
I20250109 20:58:13.956964 17466 master.cc:559] Master@127.17.14.190:37891 shutdown complete.
I20250109 20:58:13.970587 17466 master.cc:537] Master@127.17.14.189:43243 shutting down...
I20250109 20:58:13.985898 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 2 LEADER]: Raft consensus shutting down.
I20250109 20:58:13.986603 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:13.986873 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P 964362e3bdcc441c8315934ab5e26c98: stopping tablet replica
I20250109 20:58:14.004644 17466 master.cc:559] Master@127.17.14.189:43243 shutdown complete.
I20250109 20:58:14.016058 17466 master.cc:537] Master@127.17.14.188:34437 shutting down...
I20250109 20:58:14.030185 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:14.030671 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:14.030947 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P 682f1f1f507c4b7882d9605ebf01b975: stopping tablet replica
I20250109 20:58:14.048194 17466 master.cc:559] Master@127.17.14.188:34437 shutdown complete.
I20250109 20:58:14.068197 17466 test_util.cc:181] -----------------------------------------------
I20250109 20:58:14.068408 17466 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736456270445890-17466-0
[ FAILED ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (5897 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20250109 20:58:14.071143 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:37215
I20250109 20:58:14.072103 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:14.077548 19103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:14.078431 19104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:14.079802 19106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:14.080415 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:14.081123 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:14.081290 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:14.081423 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456294081410 us; error 0 us; skew 500 ppm
I20250109 20:58:14.081874 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:14.084087 17466 webserver.cc:458] Webserver started at http://127.17.14.190:46089/ using document root <none> and password file <none>
I20250109 20:58:14.084451 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:14.084581 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:14.084785 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:14.085821 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "6068266261fe4e40a18c8c41ef47c0af"
format_stamp: "Formatted at 2025-01-09 20:58:14 on dist-test-slave-j54z"
I20250109 20:58:14.089877 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250109 20:58:14.092777 19111 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:14.093510 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 20:58:14.093742 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "6068266261fe4e40a18c8c41ef47c0af"
format_stamp: "Formatted at 2025-01-09 20:58:14 on dist-test-slave-j54z"
I20250109 20:58:14.093976 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:14.106014 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:14.107065 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:14.141758 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:37215
I20250109 20:58:14.141853 19162 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:37215 every 8 connection(s)
I20250109 20:58:14.145330 19163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:14.154567 19163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af: Bootstrap starting.
I20250109 20:58:14.158654 19163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:14.162240 19163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af: No bootstrap required, opened a new log
I20250109 20:58:14.163991 19163 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6068266261fe4e40a18c8c41ef47c0af" member_type: VOTER }
I20250109 20:58:14.164327 19163 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:14.164532 19163 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6068266261fe4e40a18c8c41ef47c0af, State: Initialized, Role: FOLLOWER
I20250109 20:58:14.165053 19163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6068266261fe4e40a18c8c41ef47c0af" member_type: VOTER }
I20250109 20:58:14.165438 19163 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:58:14.165647 19163 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:58:14.165877 19163 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:14.169672 19163 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6068266261fe4e40a18c8c41ef47c0af" member_type: VOTER }
I20250109 20:58:14.170128 19163 leader_election.cc:304] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6068266261fe4e40a18c8c41ef47c0af; no voters:
I20250109 20:58:14.171140 19163 leader_election.cc:290] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:58:14.171499 19166 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:14.172803 19166 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [term 1 LEADER]: Becoming Leader. State: Replica: 6068266261fe4e40a18c8c41ef47c0af, State: Running, Role: LEADER
I20250109 20:58:14.173415 19166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6068266261fe4e40a18c8c41ef47c0af" member_type: VOTER }
I20250109 20:58:14.173820 19163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:58:14.176726 19167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6068266261fe4e40a18c8c41ef47c0af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6068266261fe4e40a18c8c41ef47c0af" member_type: VOTER } }
I20250109 20:58:14.176802 19168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6068266261fe4e40a18c8c41ef47c0af. Latest consensus state: current_term: 1 leader_uuid: "6068266261fe4e40a18c8c41ef47c0af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6068266261fe4e40a18c8c41ef47c0af" member_type: VOTER } }
I20250109 20:58:14.177465 19167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [sys.catalog]: This master's current role is: LEADER
I20250109 20:58:14.177721 19168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [sys.catalog]: This master's current role is: LEADER
I20250109 20:58:14.185959 19172 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:58:14.190876 19172 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:58:14.191736 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 20:58:14.198526 19172 catalog_manager.cc:1348] Generated new cluster ID: 11f9b764a196417bbd71935a1480bb2a
I20250109 20:58:14.198717 19172 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:58:14.214475 19172 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 20:58:14.215574 19172 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:58:14.230942 19172 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af: Generated new TSK 0
I20250109 20:58:14.231463 19172 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:58:14.257946 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:14.264029 19184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:14.265352 19185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:14.266847 17466 server_base.cc:1034] running on GCE node
W20250109 20:58:14.268112 19187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:14.268821 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:14.269004 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:14.269140 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456294269128 us; error 0 us; skew 500 ppm
I20250109 20:58:14.269572 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:14.271688 17466 webserver.cc:458] Webserver started at http://127.17.14.129:36719/ using document root <none> and password file <none>
I20250109 20:58:14.272078 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:14.272222 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:14.272414 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:14.273365 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-0-root/instance:
uuid: "d1f39ff296d243f4b84ab3926f931d7f"
format_stamp: "Formatted at 2025-01-09 20:58:14 on dist-test-slave-j54z"
I20250109 20:58:14.277624 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250109 20:58:14.280567 19192 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:14.281214 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:58:14.281455 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "d1f39ff296d243f4b84ab3926f931d7f"
format_stamp: "Formatted at 2025-01-09 20:58:14 on dist-test-slave-j54z"
I20250109 20:58:14.281744 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:14.292150 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:14.293223 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:14.294580 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:58:14.296795 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:58:14.296981 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:14.297204 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:58:14.297346 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:14.333992 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:35523
I20250109 20:58:14.334080 19254 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:35523 every 8 connection(s)
I20250109 20:58:14.338308 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:14.345922 19259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:14.347115 19260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:14.348680 19262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:14.350287 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:14.350414 19255 heartbeater.cc:346] Connected to a master server at 127.17.14.190:37215
I20250109 20:58:14.350785 19255 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:14.351033 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:14.351236 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:14.351377 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456294351360 us; error 0 us; skew 500 ppm
I20250109 20:58:14.351449 19255 heartbeater.cc:510] Master 127.17.14.190:37215 requested a full tablet report, sending...
I20250109 20:58:14.351873 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:14.353196 19128 ts_manager.cc:194] Registered new tserver with Master: d1f39ff296d243f4b84ab3926f931d7f (127.17.14.129:35523)
I20250109 20:58:14.354563 17466 webserver.cc:458] Webserver started at http://127.17.14.130:46881/ using document root <none> and password file <none>
I20250109 20:58:14.354828 19128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46290
I20250109 20:58:14.354995 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:14.355226 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:14.355456 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:14.356468 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-1-root/instance:
uuid: "2cb65301844c436d8e3674715c5cc230"
format_stamp: "Formatted at 2025-01-09 20:58:14 on dist-test-slave-j54z"
I20250109 20:58:14.360694 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 20:58:14.363742 19267 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:14.364429 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 20:58:14.364663 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "2cb65301844c436d8e3674715c5cc230"
format_stamp: "Formatted at 2025-01-09 20:58:14 on dist-test-slave-j54z"
I20250109 20:58:14.364905 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:14.379855 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:14.380934 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:14.382237 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:58:14.384467 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:58:14.384642 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:14.384845 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:58:14.384984 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:14.420641 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:43367
I20250109 20:58:14.420737 19329 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:43367 every 8 connection(s)
I20250109 20:58:14.425244 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:14.432114 19333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:14.433790 19334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:14.435299 17466 server_base.cc:1034] running on GCE node
W20250109 20:58:14.436482 19336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:14.437199 19330 heartbeater.cc:346] Connected to a master server at 127.17.14.190:37215
I20250109 20:58:14.437353 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250109 20:58:14.437526 19330 heartbeater.cc:463] Registering TS with master...
W20250109 20:58:14.437635 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:14.437866 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456294437850 us; error 0 us; skew 500 ppm
I20250109 20:58:14.438292 19330 heartbeater.cc:510] Master 127.17.14.190:37215 requested a full tablet report, sending...
I20250109 20:58:14.438388 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:14.440387 19128 ts_manager.cc:194] Registered new tserver with Master: 2cb65301844c436d8e3674715c5cc230 (127.17.14.130:43367)
I20250109 20:58:14.441129 17466 webserver.cc:458] Webserver started at http://127.17.14.131:33065/ using document root <none> and password file <none>
I20250109 20:58:14.441741 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:14.441959 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:14.442142 19128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46298
I20250109 20:58:14.442260 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:14.443447 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-2-root/instance:
uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c"
format_stamp: "Formatted at 2025-01-09 20:58:14 on dist-test-slave-j54z"
I20250109 20:58:14.447453 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250109 20:58:14.450246 19341 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:14.450887 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:58:14.451155 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c"
format_stamp: "Formatted at 2025-01-09 20:58:14 on dist-test-slave-j54z"
I20250109 20:58:14.451399 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:14.476909 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:14.478034 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:14.479342 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:58:14.481328 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:58:14.481525 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:14.481724 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:58:14.481865 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:14.517634 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:42473
I20250109 20:58:14.517717 19403 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:42473 every 8 connection(s)
I20250109 20:58:14.528863 19404 heartbeater.cc:346] Connected to a master server at 127.17.14.190:37215
I20250109 20:58:14.529172 19404 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:14.529812 19404 heartbeater.cc:510] Master 127.17.14.190:37215 requested a full tablet report, sending...
I20250109 20:58:14.531471 19128 ts_manager.cc:194] Registered new tserver with Master: 78e3b9f7d0004e6bac8845e8cdbc9a6c (127.17.14.131:42473)
I20250109 20:58:14.532500 17466 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011983919s
I20250109 20:58:14.532855 19128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46314
I20250109 20:58:15.357124 19255 heartbeater.cc:502] Master 127.17.14.190:37215 was elected leader, sending a full tablet report...
I20250109 20:58:15.444298 19330 heartbeater.cc:502] Master 127.17.14.190:37215 was elected leader, sending a full tablet report...
I20250109 20:58:15.535123 19404 heartbeater.cc:502] Master 127.17.14.190:37215 was elected leader, sending a full tablet report...
I20250109 20:58:15.564488 17466 test_util.cc:274] Using random seed: 1017791661
I20250109 20:58:15.584087 19128 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46328:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:58:15.585716 19128 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:58:15.622563 19220 tablet_service.cc:1467] Processing CreateTablet for tablet 160a529ef1304440987873353ccb1dea (DEFAULT_TABLE table=test-workload [id=aa89ebce7c61477ba1ff82ca89284a70]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250109 20:58:15.623014 19219 tablet_service.cc:1467] Processing CreateTablet for tablet 794ea35e02e24b79a994932948e2db54 (DEFAULT_TABLE table=test-workload [id=aa89ebce7c61477ba1ff82ca89284a70]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250109 20:58:15.623894 19220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160a529ef1304440987873353ccb1dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:15.624493 19219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 794ea35e02e24b79a994932948e2db54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:15.624444 19369 tablet_service.cc:1467] Processing CreateTablet for tablet 160a529ef1304440987873353ccb1dea (DEFAULT_TABLE table=test-workload [id=aa89ebce7c61477ba1ff82ca89284a70]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250109 20:58:15.625788 19369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160a529ef1304440987873353ccb1dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:15.625799 19368 tablet_service.cc:1467] Processing CreateTablet for tablet 794ea35e02e24b79a994932948e2db54 (DEFAULT_TABLE table=test-workload [id=aa89ebce7c61477ba1ff82ca89284a70]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250109 20:58:15.627979 19368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 794ea35e02e24b79a994932948e2db54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:15.628381 19295 tablet_service.cc:1467] Processing CreateTablet for tablet 160a529ef1304440987873353ccb1dea (DEFAULT_TABLE table=test-workload [id=aa89ebce7c61477ba1ff82ca89284a70]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250109 20:58:15.629549 19295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160a529ef1304440987873353ccb1dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:15.634430 19294 tablet_service.cc:1467] Processing CreateTablet for tablet 794ea35e02e24b79a994932948e2db54 (DEFAULT_TABLE table=test-workload [id=aa89ebce7c61477ba1ff82ca89284a70]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250109 20:58:15.635589 19294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 794ea35e02e24b79a994932948e2db54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:15.646811 19424 tablet_bootstrap.cc:492] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f: Bootstrap starting.
I20250109 20:58:15.653771 19424 tablet_bootstrap.cc:654] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:15.648110 19425 tablet_bootstrap.cc:492] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c: Bootstrap starting.
I20250109 20:58:15.659582 19425 tablet_bootstrap.cc:654] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:15.661720 19426 tablet_bootstrap.cc:492] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230: Bootstrap starting.
I20250109 20:58:15.664641 19424 tablet_bootstrap.cc:492] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f: No bootstrap required, opened a new log
I20250109 20:58:15.665042 19424 ts_tablet_manager.cc:1397] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.004s
I20250109 20:58:15.666692 19426 tablet_bootstrap.cc:654] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:15.667439 19425 tablet_bootstrap.cc:492] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c: No bootstrap required, opened a new log
I20250109 20:58:15.667871 19425 ts_tablet_manager.cc:1397] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.009s
I20250109 20:58:15.667763 19424 raft_consensus.cc:357] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } }
I20250109 20:58:15.668593 19424 raft_consensus.cc:383] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:15.668941 19424 raft_consensus.cc:738] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f39ff296d243f4b84ab3926f931d7f, State: Initialized, Role: FOLLOWER
I20250109 20:58:15.669703 19424 consensus_queue.cc:260] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } }
I20250109 20:58:15.670687 19426 tablet_bootstrap.cc:492] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230: No bootstrap required, opened a new log
I20250109 20:58:15.670521 19425 raft_consensus.cc:357] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } }
I20250109 20:58:15.671057 19426 ts_tablet_manager.cc:1397] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20250109 20:58:15.671409 19425 raft_consensus.cc:383] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:15.671806 19425 raft_consensus.cc:738] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e3b9f7d0004e6bac8845e8cdbc9a6c, State: Initialized, Role: FOLLOWER
I20250109 20:58:15.672449 19425 consensus_queue.cc:260] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } }
I20250109 20:58:15.673436 19426 raft_consensus.cc:357] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } }
I20250109 20:58:15.674090 19426 raft_consensus.cc:383] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:15.674388 19426 raft_consensus.cc:738] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb65301844c436d8e3674715c5cc230, State: Initialized, Role: FOLLOWER
I20250109 20:58:15.674453 19424 ts_tablet_manager.cc:1428] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20250109 20:58:15.675433 19424 tablet_bootstrap.cc:492] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f: Bootstrap starting.
I20250109 20:58:15.675242 19426 consensus_queue.cc:260] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } }
I20250109 20:58:15.679669 19426 ts_tablet_manager.cc:1428] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250109 20:58:15.680634 19426 tablet_bootstrap.cc:492] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230: Bootstrap starting.
I20250109 20:58:15.685657 19426 tablet_bootstrap.cc:654] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:15.688208 19424 tablet_bootstrap.cc:654] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:15.686488 19425 ts_tablet_manager.cc:1428] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c: Time spent starting tablet: real 0.018s user 0.006s sys 0.007s
I20250109 20:58:15.690474 19426 tablet_bootstrap.cc:492] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230: No bootstrap required, opened a new log
I20250109 20:58:15.690501 19425 tablet_bootstrap.cc:492] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c: Bootstrap starting.
I20250109 20:58:15.691115 19426 ts_tablet_manager.cc:1397] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20250109 20:58:15.693133 19426 raft_consensus.cc:357] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } }
I20250109 20:58:15.693629 19426 raft_consensus.cc:383] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:15.693828 19426 raft_consensus.cc:738] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb65301844c436d8e3674715c5cc230, State: Initialized, Role: FOLLOWER
I20250109 20:58:15.694326 19426 consensus_queue.cc:260] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } }
I20250109 20:58:15.695870 19426 ts_tablet_manager.cc:1428] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:58:15.699349 19424 tablet_bootstrap.cc:492] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f: No bootstrap required, opened a new log
I20250109 20:58:15.699816 19424 ts_tablet_manager.cc:1397] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.008s
I20250109 20:58:15.702438 19424 raft_consensus.cc:357] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } }
I20250109 20:58:15.703281 19424 raft_consensus.cc:383] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:15.703603 19424 raft_consensus.cc:738] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f39ff296d243f4b84ab3926f931d7f, State: Initialized, Role: FOLLOWER
I20250109 20:58:15.704260 19424 consensus_queue.cc:260] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } }
I20250109 20:58:15.705814 19425 tablet_bootstrap.cc:654] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:15.706336 19424 ts_tablet_manager.cc:1428] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250109 20:58:15.709600 19432 raft_consensus.cc:491] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:15.710052 19425 tablet_bootstrap.cc:492] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c: No bootstrap required, opened a new log
I20250109 20:58:15.710633 19425 ts_tablet_manager.cc:1397] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20250109 20:58:15.710070 19432 raft_consensus.cc:513] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } }
I20250109 20:58:15.712904 19425 raft_consensus.cc:357] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } }
I20250109 20:58:15.713693 19425 raft_consensus.cc:383] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:15.714004 19425 raft_consensus.cc:738] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e3b9f7d0004e6bac8845e8cdbc9a6c, State: Initialized, Role: FOLLOWER
I20250109 20:58:15.714704 19432 leader_election.cc:290] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f39ff296d243f4b84ab3926f931d7f (127.17.14.129:35523), 78e3b9f7d0004e6bac8845e8cdbc9a6c (127.17.14.131:42473)
I20250109 20:58:15.714744 19425 consensus_queue.cc:260] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } }
I20250109 20:58:15.717654 19425 ts_tablet_manager.cc:1428] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20250109 20:58:15.722857 19230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794ea35e02e24b79a994932948e2db54" candidate_uuid: "2cb65301844c436d8e3674715c5cc230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f39ff296d243f4b84ab3926f931d7f" is_pre_election: true
I20250109 20:58:15.723563 19230 raft_consensus.cc:2463] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb65301844c436d8e3674715c5cc230 in term 0.
I20250109 20:58:15.724562 19270 leader_election.cc:304] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb65301844c436d8e3674715c5cc230, d1f39ff296d243f4b84ab3926f931d7f; no voters:
I20250109 20:58:15.725165 19432 raft_consensus.cc:2798] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:58:15.725438 19432 raft_consensus.cc:491] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:58:15.725739 19432 raft_consensus.cc:3054] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:15.727890 19379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794ea35e02e24b79a994932948e2db54" candidate_uuid: "2cb65301844c436d8e3674715c5cc230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" is_pre_election: true
I20250109 20:58:15.728404 19379 raft_consensus.cc:2463] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb65301844c436d8e3674715c5cc230 in term 0.
I20250109 20:58:15.730753 19432 raft_consensus.cc:513] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } }
I20250109 20:58:15.732192 19432 leader_election.cc:290] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [CANDIDATE]: Term 1 election: Requested vote from peers d1f39ff296d243f4b84ab3926f931d7f (127.17.14.129:35523), 78e3b9f7d0004e6bac8845e8cdbc9a6c (127.17.14.131:42473)
I20250109 20:58:15.732674 19432 raft_consensus.cc:491] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:58:15.732829 19230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794ea35e02e24b79a994932948e2db54" candidate_uuid: "2cb65301844c436d8e3674715c5cc230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f39ff296d243f4b84ab3926f931d7f"
I20250109 20:58:15.733060 19432 raft_consensus.cc:513] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } }
I20250109 20:58:15.733130 19379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794ea35e02e24b79a994932948e2db54" candidate_uuid: "2cb65301844c436d8e3674715c5cc230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c"
I20250109 20:58:15.733412 19230 raft_consensus.cc:3054] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:15.733783 19379 raft_consensus.cc:3054] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:15.734642 19432 leader_election.cc:290] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78e3b9f7d0004e6bac8845e8cdbc9a6c (127.17.14.131:42473), d1f39ff296d243f4b84ab3926f931d7f (127.17.14.129:35523)
I20250109 20:58:15.735215 19378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160a529ef1304440987873353ccb1dea" candidate_uuid: "2cb65301844c436d8e3674715c5cc230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" is_pre_election: true
I20250109 20:58:15.735694 19378 raft_consensus.cc:2463] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb65301844c436d8e3674715c5cc230 in term 0.
I20250109 20:58:15.736917 19269 leader_election.cc:304] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb65301844c436d8e3674715c5cc230, 78e3b9f7d0004e6bac8845e8cdbc9a6c; no voters:
I20250109 20:58:15.737555 19432 raft_consensus.cc:2798] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:58:15.737321 19229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160a529ef1304440987873353ccb1dea" candidate_uuid: "2cb65301844c436d8e3674715c5cc230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f39ff296d243f4b84ab3926f931d7f" is_pre_election: true
I20250109 20:58:15.737864 19432 raft_consensus.cc:491] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:58:15.738161 19432 raft_consensus.cc:3054] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:15.738085 19229 raft_consensus.cc:2463] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb65301844c436d8e3674715c5cc230 in term 0.
I20250109 20:58:15.740044 19379 raft_consensus.cc:2463] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb65301844c436d8e3674715c5cc230 in term 1.
I20250109 20:58:15.741106 19269 leader_election.cc:304] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb65301844c436d8e3674715c5cc230, 78e3b9f7d0004e6bac8845e8cdbc9a6c; no voters:
I20250109 20:58:15.742338 19230 raft_consensus.cc:2463] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb65301844c436d8e3674715c5cc230 in term 1.
I20250109 20:58:15.742928 19438 raft_consensus.cc:2798] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:15.744177 19438 raft_consensus.cc:695] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 1 LEADER]: Becoming Leader. State: Replica: 2cb65301844c436d8e3674715c5cc230, State: Running, Role: LEADER
I20250109 20:58:15.744930 19432 raft_consensus.cc:513] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } }
I20250109 20:58:15.744997 19438 consensus_queue.cc:237] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } }
I20250109 20:58:15.746845 19432 leader_election.cc:290] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [CANDIDATE]: Term 1 election: Requested vote from peers 78e3b9f7d0004e6bac8845e8cdbc9a6c (127.17.14.131:42473), d1f39ff296d243f4b84ab3926f931d7f (127.17.14.129:35523)
I20250109 20:58:15.747455 19379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160a529ef1304440987873353ccb1dea" candidate_uuid: "2cb65301844c436d8e3674715c5cc230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c"
I20250109 20:58:15.747905 19379 raft_consensus.cc:3054] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:15.747771 19230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160a529ef1304440987873353ccb1dea" candidate_uuid: "2cb65301844c436d8e3674715c5cc230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f39ff296d243f4b84ab3926f931d7f"
I20250109 20:58:15.748514 19230 raft_consensus.cc:3054] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:15.753295 19379 raft_consensus.cc:2463] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb65301844c436d8e3674715c5cc230 in term 1.
I20250109 20:58:15.752825 19128 catalog_manager.cc:5526] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cb65301844c436d8e3674715c5cc230 (127.17.14.130). New cstate: current_term: 1 leader_uuid: "2cb65301844c436d8e3674715c5cc230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20250109 20:58:15.753633 19230 raft_consensus.cc:2463] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb65301844c436d8e3674715c5cc230 in term 1.
I20250109 20:58:15.754139 19269 leader_election.cc:304] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb65301844c436d8e3674715c5cc230, 78e3b9f7d0004e6bac8845e8cdbc9a6c; no voters:
I20250109 20:58:15.754812 19438 raft_consensus.cc:2798] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:15.755177 19438 raft_consensus.cc:695] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 1 LEADER]: Becoming Leader. State: Replica: 2cb65301844c436d8e3674715c5cc230, State: Running, Role: LEADER
I20250109 20:58:15.755760 19438 consensus_queue.cc:237] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } }
I20250109 20:58:15.764443 19128 catalog_manager.cc:5526] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cb65301844c436d8e3674715c5cc230 (127.17.14.130). New cstate: current_term: 1 leader_uuid: "2cb65301844c436d8e3674715c5cc230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } health_report { overall_health: HEALTHY } } }
I20250109 20:58:16.175668 19438 consensus_queue.cc:1035] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:16.186954 19432 consensus_queue.cc:1035] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:16.188561 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:16.205899 19305 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "160a529ef1304440987873353ccb1dea"
dest_uuid: "2cb65301844c436d8e3674715c5cc230"
mode: GRACEFUL
new_leader_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c"
from {username='slave'} at 127.0.0.1:56670
I20250109 20:58:16.206591 19305 raft_consensus.cc:604] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 1 LEADER]: Received request to transfer leadership to TS 78e3b9f7d0004e6bac8845e8cdbc9a6c
I20250109 20:58:16.208055 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250109 20:58:16.208386 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:16.230109 19432 consensus_queue.cc:1035] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:16.239935 19439 consensus_queue.cc:1035] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:58:16.744745 19452 raft_consensus.cc:988] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230: : Instructing follower 78e3b9f7d0004e6bac8845e8cdbc9a6c to start an election
I20250109 20:58:16.745242 19453 raft_consensus.cc:1076] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 1 LEADER]: Signalling peer 78e3b9f7d0004e6bac8845e8cdbc9a6c to start an election
I20250109 20:58:16.746481 19379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "160a529ef1304440987873353ccb1dea"
dest_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c"
from {username='slave'} at 127.0.0.1:39622
I20250109 20:58:16.747069 19379 raft_consensus.cc:491] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250109 20:58:16.747414 19379 raft_consensus.cc:3054] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:16.751645 19379 raft_consensus.cc:513] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } }
I20250109 20:58:16.753345 19379 leader_election.cc:290] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [CANDIDATE]: Term 2 election: Requested vote from peers d1f39ff296d243f4b84ab3926f931d7f (127.17.14.129:35523), 2cb65301844c436d8e3674715c5cc230 (127.17.14.130:43367)
I20250109 20:58:16.773054 19230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160a529ef1304440987873353ccb1dea" candidate_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1f39ff296d243f4b84ab3926f931d7f"
I20250109 20:58:16.773645 19230 raft_consensus.cc:3054] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:16.774963 19305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160a529ef1304440987873353ccb1dea" candidate_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2cb65301844c436d8e3674715c5cc230"
I20250109 20:58:16.775475 19305 raft_consensus.cc:3049] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 1 LEADER]: Stepping down as leader of term 1
I20250109 20:58:16.775796 19305 raft_consensus.cc:738] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2cb65301844c436d8e3674715c5cc230, State: Running, Role: LEADER
I20250109 20:58:16.776494 19305 consensus_queue.cc:260] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } }
I20250109 20:58:16.777407 19305 raft_consensus.cc:3054] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:58:16.778463 19230 raft_consensus.cc:2463] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e3b9f7d0004e6bac8845e8cdbc9a6c in term 2.
I20250109 20:58:16.779358 19344 leader_election.cc:304] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78e3b9f7d0004e6bac8845e8cdbc9a6c, d1f39ff296d243f4b84ab3926f931d7f; no voters:
I20250109 20:58:16.780735 19458 raft_consensus.cc:2798] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 2 FOLLOWER]: Leader election won for term 2
I20250109 20:58:16.781944 19458 raft_consensus.cc:695] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 2 LEADER]: Becoming Leader. State: Replica: 78e3b9f7d0004e6bac8845e8cdbc9a6c, State: Running, Role: LEADER
I20250109 20:58:16.781960 19305 raft_consensus.cc:2463] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e3b9f7d0004e6bac8845e8cdbc9a6c in term 2.
I20250109 20:58:16.782914 19458 consensus_queue.cc:237] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } }
I20250109 20:58:16.788545 19128 catalog_manager.cc:5526] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c reported cstate change: term changed from 1 to 2, leader changed from 2cb65301844c436d8e3674715c5cc230 (127.17.14.130) to 78e3b9f7d0004e6bac8845e8cdbc9a6c (127.17.14.131). New cstate: current_term: 2 leader_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } health_report { overall_health: UNKNOWN } } }
I20250109 20:58:17.261196 19230 raft_consensus.cc:1270] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 2 FOLLOWER]: Refusing update from remote peer 78e3b9f7d0004e6bac8845e8cdbc9a6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:58:17.262346 19458 consensus_queue.cc:1035] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:17.270929 19305 raft_consensus.cc:1270] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Refusing update from remote peer 78e3b9f7d0004e6bac8845e8cdbc9a6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:58:17.273798 19467 consensus_queue.cc:1035] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:58:18.209229 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:18.211160 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:18.211477 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:20.212260 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:20.214041 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:20.214339 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:22.215263 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:22.217196 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:22.217542 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:24.218398 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:24.220224 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:24.220544 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:26.221382 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:26.223274 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:26.223573 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:26.860386 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:26.868052 19480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:26.869336 19481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:26.870014 19483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:26.870386 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:26.871280 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:26.871479 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:26.871645 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456306871630 us; error 0 us; skew 500 ppm
I20250109 20:58:26.872100 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:26.879092 17466 webserver.cc:458] Webserver started at http://127.17.14.132:35443/ using document root <none> and password file <none>
I20250109 20:58:26.879583 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:26.879801 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:26.880115 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:26.881510 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-3-root/instance:
uuid: "1f1ba3d2da294218bf6198d35bf2b23e"
format_stamp: "Formatted at 2025-01-09 20:58:26 on dist-test-slave-j54z"
I20250109 20:58:26.886158 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250109 20:58:26.889359 19488 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:26.890125 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 20:58:26.890388 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-3-root
uuid: "1f1ba3d2da294218bf6198d35bf2b23e"
format_stamp: "Formatted at 2025-01-09 20:58:26 on dist-test-slave-j54z"
I20250109 20:58:26.890655 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:26.904410 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:26.905714 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:26.907272 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:58:26.909893 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:58:26.910097 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:26.910323 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:58:26.910487 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:26.950309 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.132:43125
I20250109 20:58:26.950412 19550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.132:43125 every 8 connection(s)
I20250109 20:58:26.964263 19551 heartbeater.cc:346] Connected to a master server at 127.17.14.190:37215
I20250109 20:58:26.964604 19551 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:26.965338 19551 heartbeater.cc:510] Master 127.17.14.190:37215 requested a full tablet report, sending...
I20250109 20:58:26.967162 19127 ts_manager.cc:194] Registered new tserver with Master: 1f1ba3d2da294218bf6198d35bf2b23e (127.17.14.132:43125)
I20250109 20:58:26.968513 19127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54470
I20250109 20:58:27.250309 19379 consensus_queue.cc:237] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: true } }
I20250109 20:58:27.257156 19305 raft_consensus.cc:1270] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Refusing update from remote peer 78e3b9f7d0004e6bac8845e8cdbc9a6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 20:58:27.258474 19229 raft_consensus.cc:1270] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 2 FOLLOWER]: Refusing update from remote peer 78e3b9f7d0004e6bac8845e8cdbc9a6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 20:58:27.260387 19558 consensus_queue.cc:1035] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250109 20:58:27.261607 19475 consensus_queue.cc:1035] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250109 20:58:27.273373 19558 raft_consensus.cc:2949] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1f1ba3d2da294218bf6198d35bf2b23e (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: true } } }
I20250109 20:58:27.275295 19305 raft_consensus.cc:2949] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1f1ba3d2da294218bf6198d35bf2b23e (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: true } } }
W20250109 20:58:27.277042 19345 consensus_peers.cc:487] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c -> Peer 1f1ba3d2da294218bf6198d35bf2b23e (127.17.14.132:43125): Couldn't send request to peer 1f1ba3d2da294218bf6198d35bf2b23e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 160a529ef1304440987873353ccb1dea. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:58:27.276945 19229 raft_consensus.cc:2949] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1f1ba3d2da294218bf6198d35bf2b23e (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: true } } }
I20250109 20:58:27.287146 19127 catalog_manager.cc:5526] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 reported cstate change: config changed from index -1 to 3, NON_VOTER 1f1ba3d2da294218bf6198d35bf2b23e (127.17.14.132) added. New cstate: current_term: 2 leader_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: true } } }
I20250109 20:58:27.788733 19569 ts_tablet_manager.cc:927] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e: Initiating tablet copy from peer 78e3b9f7d0004e6bac8845e8cdbc9a6c (127.17.14.131:42473)
I20250109 20:58:27.790544 19569 tablet_copy_client.cc:323] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.131:42473
I20250109 20:58:27.801345 19389 tablet_copy_service.cc:140] P 78e3b9f7d0004e6bac8845e8cdbc9a6c: Received BeginTabletCopySession request for tablet 160a529ef1304440987873353ccb1dea from peer 1f1ba3d2da294218bf6198d35bf2b23e ({username='slave'} at 127.0.0.1:39724)
I20250109 20:58:27.801920 19389 tablet_copy_service.cc:161] P 78e3b9f7d0004e6bac8845e8cdbc9a6c: Beginning new tablet copy session on tablet 160a529ef1304440987873353ccb1dea from peer 1f1ba3d2da294218bf6198d35bf2b23e at {username='slave'} at 127.0.0.1:39724: session id = 1f1ba3d2da294218bf6198d35bf2b23e-160a529ef1304440987873353ccb1dea
I20250109 20:58:27.808952 19389 tablet_copy_source_session.cc:215] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 20:58:27.811566 19569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160a529ef1304440987873353ccb1dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:27.823261 19569 tablet_copy_client.cc:806] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e: tablet copy: Starting download of 0 data blocks...
I20250109 20:58:27.823771 19569 tablet_copy_client.cc:670] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e: tablet copy: Starting download of 1 WAL segments...
I20250109 20:58:27.826951 19569 tablet_copy_client.cc:538] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 20:58:27.833922 19569 tablet_bootstrap.cc:492] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e: Bootstrap starting.
I20250109 20:58:27.851855 19569 tablet_bootstrap.cc:492] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:58:27.852730 19569 tablet_bootstrap.cc:492] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e: Bootstrap complete.
I20250109 20:58:27.853253 19569 ts_tablet_manager.cc:1397] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20250109 20:58:27.855648 19569 raft_consensus.cc:357] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: true } }
I20250109 20:58:27.856395 19569 raft_consensus.cc:738] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1f1ba3d2da294218bf6198d35bf2b23e, State: Initialized, Role: LEARNER
I20250109 20:58:27.857028 19569 consensus_queue.cc:260] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: true } }
I20250109 20:58:27.859033 19551 heartbeater.cc:502] Master 127.17.14.190:37215 was elected leader, sending a full tablet report...
I20250109 20:58:27.859629 19569 ts_tablet_manager.cc:1428] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
I20250109 20:58:27.861212 19389 tablet_copy_service.cc:342] P 78e3b9f7d0004e6bac8845e8cdbc9a6c: Request end of tablet copy session 1f1ba3d2da294218bf6198d35bf2b23e-160a529ef1304440987873353ccb1dea received from {username='slave'} at 127.0.0.1:39724
I20250109 20:58:27.861680 19389 tablet_copy_service.cc:434] P 78e3b9f7d0004e6bac8845e8cdbc9a6c: ending tablet copy session 1f1ba3d2da294218bf6198d35bf2b23e-160a529ef1304440987873353ccb1dea on tablet 160a529ef1304440987873353ccb1dea with peer 1f1ba3d2da294218bf6198d35bf2b23e
I20250109 20:58:28.224373 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:28.226336 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:28.226661 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:28.258824 19526 raft_consensus.cc:1212] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250109 20:58:28.684957 19577 raft_consensus.cc:1059] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c: attempting to promote NON_VOTER 1f1ba3d2da294218bf6198d35bf2b23e to VOTER
I20250109 20:58:28.687122 19577 consensus_queue.cc:237] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false } }
I20250109 20:58:28.692394 19305 raft_consensus.cc:1270] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Refusing update from remote peer 78e3b9f7d0004e6bac8845e8cdbc9a6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 20:58:28.692844 19526 raft_consensus.cc:1270] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e [term 2 LEARNER]: Refusing update from remote peer 78e3b9f7d0004e6bac8845e8cdbc9a6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 20:58:28.694041 19229 raft_consensus.cc:1270] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 2 FOLLOWER]: Refusing update from remote peer 78e3b9f7d0004e6bac8845e8cdbc9a6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 20:58:28.694342 19558 consensus_queue.cc:1035] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 20:58:28.695181 19577 consensus_queue.cc:1035] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 20:58:28.696261 19576 consensus_queue.cc:1035] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250109 20:58:28.714860 19558 raft_consensus.cc:2949] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1f1ba3d2da294218bf6198d35bf2b23e (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false } } }
I20250109 20:58:28.716601 19526 raft_consensus.cc:2949] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1f1ba3d2da294218bf6198d35bf2b23e (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false } } }
I20250109 20:58:28.717520 19305 raft_consensus.cc:2949] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1f1ba3d2da294218bf6198d35bf2b23e (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false } } }
I20250109 20:58:28.726568 19229 raft_consensus.cc:2949] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1f1ba3d2da294218bf6198d35bf2b23e (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false } } }
I20250109 20:58:28.730538 19128 catalog_manager.cc:5526] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 reported cstate change: config changed from index 3 to 4, 1f1ba3d2da294218bf6198d35bf2b23e (127.17.14.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } attrs { replace: true } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false } } }
I20250109 20:58:28.754954 19378 consensus_queue.cc:237] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false } }
I20250109 20:58:28.759114 19526 raft_consensus.cc:1270] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e [term 2 FOLLOWER]: Refusing update from remote peer 78e3b9f7d0004e6bac8845e8cdbc9a6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 20:58:28.760434 19305 raft_consensus.cc:1270] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Refusing update from remote peer 78e3b9f7d0004e6bac8845e8cdbc9a6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 20:58:28.760581 19582 consensus_queue.cc:1035] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250109 20:58:28.761655 19576 consensus_queue.cc:1035] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:58:28.767354 19582 raft_consensus.cc:2949] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d1f39ff296d243f4b84ab3926f931d7f (127.17.14.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false } } }
I20250109 20:58:28.769321 19526 raft_consensus.cc:2949] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d1f39ff296d243f4b84ab3926f931d7f (127.17.14.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false } } }
I20250109 20:58:28.769801 19305 raft_consensus.cc:2949] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d1f39ff296d243f4b84ab3926f931d7f (127.17.14.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false } } }
I20250109 20:58:28.779479 19125 catalog_manager.cc:5526] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e reported cstate change: config changed from index 4 to 5, VOTER d1f39ff296d243f4b84ab3926f931d7f (127.17.14.129) evicted. New cstate: current_term: 2 leader_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false } } }
I20250109 20:58:28.780719 19113 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 160a529ef1304440987873353ccb1dea with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 20:58:28.789606 19219 tablet_service.cc:1514] Processing DeleteTablet for tablet 160a529ef1304440987873353ccb1dea with delete_type TABLET_DATA_TOMBSTONED (TS d1f39ff296d243f4b84ab3926f931d7f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33784
I20250109 20:58:28.791978 19588 tablet_replica.cc:331] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f: stopping tablet replica
I20250109 20:58:28.792908 19588 raft_consensus.cc:2238] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:28.793689 19588 raft_consensus.cc:2267] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:28.796554 19588 ts_tablet_manager.cc:1905] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250109 20:58:28.806259 19588 ts_tablet_manager.cc:1918] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250109 20:58:28.806571 19588 log.cc:1198] T 160a529ef1304440987873353ccb1dea P d1f39ff296d243f4b84ab3926f931d7f: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736456270445890-17466-0/minicluster-data/ts-0-root/wals/160a529ef1304440987873353ccb1dea
I20250109 20:58:28.807976 19114 catalog_manager.cc:4872] TS d1f39ff296d243f4b84ab3926f931d7f (127.17.14.129:35523): tablet 160a529ef1304440987873353ccb1dea (table test-workload [id=aa89ebce7c61477ba1ff82ca89284a70]) successfully deleted
I20250109 20:58:30.227648 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:30.230015 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:30.230314 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:32.847283 19591 raft_consensus.cc:491] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 78e3b9f7d0004e6bac8845e8cdbc9a6c)
I20250109 20:58:32.848472 19591 raft_consensus.cc:513] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43125 } attrs { promote: false } }
I20250109 20:58:32.852488 19591 leader_election.cc:290] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78e3b9f7d0004e6bac8845e8cdbc9a6c (127.17.14.131:42473), 1f1ba3d2da294218bf6198d35bf2b23e (127.17.14.132:43125)
I20250109 20:58:32.896577 19526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160a529ef1304440987873353ccb1dea" candidate_uuid: "2cb65301844c436d8e3674715c5cc230" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1f1ba3d2da294218bf6198d35bf2b23e" is_pre_election: true
I20250109 20:58:32.899686 19595 raft_consensus.cc:491] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2cb65301844c436d8e3674715c5cc230)
I20250109 20:58:32.900310 19595 raft_consensus.cc:513] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } }
I20250109 20:58:32.902947 19595 leader_election.cc:290] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cb65301844c436d8e3674715c5cc230 (127.17.14.130:43367), 78e3b9f7d0004e6bac8845e8cdbc9a6c (127.17.14.131:42473)
I20250109 20:58:32.921624 19305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794ea35e02e24b79a994932948e2db54" candidate_uuid: "d1f39ff296d243f4b84ab3926f931d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2cb65301844c436d8e3674715c5cc230" is_pre_election: true
I20250109 20:58:32.927999 19594 raft_consensus.cc:491] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2cb65301844c436d8e3674715c5cc230)
I20250109 20:58:32.928737 19594 raft_consensus.cc:513] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f39ff296d243f4b84ab3926f931d7f" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 35523 } } peers { permanent_uuid: "2cb65301844c436d8e3674715c5cc230" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 43367 } } peers { permanent_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 42473 } }
I20250109 20:58:32.935110 19594 leader_election.cc:290] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f39ff296d243f4b84ab3926f931d7f (127.17.14.129:35523), 2cb65301844c436d8e3674715c5cc230 (127.17.14.130:43367)
I20250109 20:58:32.938386 19378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160a529ef1304440987873353ccb1dea" candidate_uuid: "2cb65301844c436d8e3674715c5cc230" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" is_pre_election: true
I20250109 20:58:32.940479 19269 leader_election.cc:304] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2cb65301844c436d8e3674715c5cc230; no voters: 1f1ba3d2da294218bf6198d35bf2b23e, 78e3b9f7d0004e6bac8845e8cdbc9a6c
I20250109 20:58:32.941797 19591 raft_consensus.cc:2743] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250109 20:58:32.948681 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:32.949369 19229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794ea35e02e24b79a994932948e2db54" candidate_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1f39ff296d243f4b84ab3926f931d7f" is_pre_election: true
I20250109 20:58:32.951390 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:32.951750 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:32.959229 19305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794ea35e02e24b79a994932948e2db54" candidate_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2cb65301844c436d8e3674715c5cc230" is_pre_election: true
I20250109 20:58:32.960762 19343 leader_election.cc:304] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78e3b9f7d0004e6bac8845e8cdbc9a6c; no voters: 2cb65301844c436d8e3674715c5cc230, d1f39ff296d243f4b84ab3926f931d7f
I20250109 20:58:32.961546 19594 raft_consensus.cc:2743] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:58:32.968107 19379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794ea35e02e24b79a994932948e2db54" candidate_uuid: "d1f39ff296d243f4b84ab3926f931d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78e3b9f7d0004e6bac8845e8cdbc9a6c" is_pre_election: true
I20250109 20:58:32.970352 19194 leader_election.cc:304] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1f39ff296d243f4b84ab3926f931d7f; no voters: 2cb65301844c436d8e3674715c5cc230, 78e3b9f7d0004e6bac8845e8cdbc9a6c
I20250109 20:58:32.971237 19595 raft_consensus.cc:2743] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 20:58:33.924011 19400 debug-util.cc:398] Leaking SignalData structure 0x7b08000ae720 after lost signal to thread 17469
W20250109 20:58:33.925160 19400 debug-util.cc:398] Leaking SignalData structure 0x7b08000a62e0 after lost signal to thread 19162
W20250109 20:58:33.926235 19400 debug-util.cc:398] Leaking SignalData structure 0x7b08000d3ae0 after lost signal to thread 19254
W20250109 20:58:33.927613 19400 debug-util.cc:398] Leaking SignalData structure 0x7b0800056ee0 after lost signal to thread 19403
W20250109 20:58:33.928465 19400 debug-util.cc:398] Leaking SignalData structure 0x7b0800101c40 after lost signal to thread 19550
I20250109 20:58:34.952733 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:34.954607 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:34.954937 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250109 20:58:36.167905 19326 debug-util.cc:398] Leaking SignalData structure 0x7b0800156740 after lost signal to thread 17469
W20250109 20:58:36.168762 19326 debug-util.cc:398] Leaking SignalData structure 0x7b0800152800 after lost signal to thread 19162
W20250109 20:58:36.169713 19326 debug-util.cc:398] Leaking SignalData structure 0x7b080008fe80 after lost signal to thread 19254
W20250109 20:58:36.170686 19326 debug-util.cc:398] Leaking SignalData structure 0x7b08000eeae0 after lost signal to thread 19329
W20250109 20:58:36.171480 19326 debug-util.cc:398] Leaking SignalData structure 0x7b0800236b80 after lost signal to thread 19403
W20250109 20:58:36.172283 19326 debug-util.cc:398] Leaking SignalData structure 0x7b0800033fc0 after lost signal to thread 19550
I20250109 20:58:37.040632 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:37.050536 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:37.050860 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250109 20:58:38.000005 19400 debug-util.cc:398] Leaking SignalData structure 0x7b080004b020 after lost signal to thread 17469
W20250109 20:58:38.000779 19400 debug-util.cc:398] Leaking SignalData structure 0x7b080005b460 after lost signal to thread 19162
W20250109 20:58:38.001703 19400 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9fe0 after lost signal to thread 19254
W20250109 20:58:38.002475 19400 debug-util.cc:398] Leaking SignalData structure 0x7b08000ae7e0 after lost signal to thread 19329
W20250109 20:58:38.003211 19400 debug-util.cc:398] Leaking SignalData structure 0x7b0800229200 after lost signal to thread 19403
W20250109 20:58:38.003990 19400 debug-util.cc:398] Leaking SignalData structure 0x7b080005b980 after lost signal to thread 19550
I20250109 20:58:39.051795 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:39.054185 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:39.054558 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:41.056080 19181 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:58:41.059199 19181 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:58:41.059551 19181 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:41.324397 17466 tablet_server.cc:178] TabletServer@127.17.14.129:0 shutting down...
I20250109 20:58:41.347318 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:58:41.348027 17466 tablet_replica.cc:331] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f: stopping tablet replica
I20250109 20:58:41.348628 17466 raft_consensus.cc:2238] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:41.349211 17466 raft_consensus.cc:2267] T 794ea35e02e24b79a994932948e2db54 P d1f39ff296d243f4b84ab3926f931d7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:41.372054 17466 tablet_server.cc:195] TabletServer@127.17.14.129:0 shutdown complete.
I20250109 20:58:41.384259 17466 tablet_server.cc:178] TabletServer@127.17.14.130:0 shutting down...
I20250109 20:58:41.404222 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:58:41.404955 17466 tablet_replica.cc:331] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230: stopping tablet replica
I20250109 20:58:41.405642 17466 raft_consensus.cc:2238] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:58:41.406534 17466 raft_consensus.cc:2267] T 794ea35e02e24b79a994932948e2db54 P 2cb65301844c436d8e3674715c5cc230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:41.408442 17466 tablet_replica.cc:331] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230: stopping tablet replica
I20250109 20:58:41.408851 17466 raft_consensus.cc:2238] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:41.409518 17466 raft_consensus.cc:2267] T 160a529ef1304440987873353ccb1dea P 2cb65301844c436d8e3674715c5cc230 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:41.446954 17466 tablet_server.cc:195] TabletServer@127.17.14.130:0 shutdown complete.
I20250109 20:58:41.459584 17466 tablet_server.cc:178] TabletServer@127.17.14.131:0 shutting down...
I20250109 20:58:41.485420 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:58:41.486095 17466 tablet_replica.cc:331] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c: stopping tablet replica
I20250109 20:58:41.486693 17466 raft_consensus.cc:2238] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:41.487159 17466 raft_consensus.cc:2267] T 794ea35e02e24b79a994932948e2db54 P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:41.489030 17466 tablet_replica.cc:331] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c: stopping tablet replica
I20250109 20:58:41.489449 17466 raft_consensus.cc:2238] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 2 LEADER]: Raft consensus shutting down.
I20250109 20:58:41.490471 17466 raft_consensus.cc:2267] T 160a529ef1304440987873353ccb1dea P 78e3b9f7d0004e6bac8845e8cdbc9a6c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:41.534675 17466 tablet_server.cc:195] TabletServer@127.17.14.131:0 shutdown complete.
I20250109 20:58:41.550434 17466 tablet_server.cc:178] TabletServer@127.17.14.132:0 shutting down...
I20250109 20:58:41.580487 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:58:41.581065 17466 tablet_replica.cc:331] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e: stopping tablet replica
I20250109 20:58:41.581609 17466 raft_consensus.cc:2238] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:58:41.582335 17466 raft_consensus.cc:2267] T 160a529ef1304440987873353ccb1dea P 1f1ba3d2da294218bf6198d35bf2b23e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:41.600817 17466 tablet_server.cc:195] TabletServer@127.17.14.132:0 shutdown complete.
I20250109 20:58:41.610545 17466 master.cc:537] Master@127.17.14.190:37215 shutting down...
I20250109 20:58:41.627882 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:58:41.628443 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:41.628778 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6068266261fe4e40a18c8c41ef47c0af: stopping tablet replica
I20250109 20:58:41.647164 17466 master.cc:559] Master@127.17.14.190:37215 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (27614 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20250109 20:58:41.685714 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:43793
I20250109 20:58:41.686861 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:41.692713 19607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:41.693857 19608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:41.694365 19610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:41.695489 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:41.696241 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:41.696437 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:41.696547 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456321696537 us; error 0 us; skew 500 ppm
I20250109 20:58:41.696979 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:41.699373 17466 webserver.cc:458] Webserver started at http://127.17.14.190:36611/ using document root <none> and password file <none>
I20250109 20:58:41.699798 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:41.699946 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:41.700146 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:41.701272 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "332a708cffb544b9b79e8b7495614180"
format_stamp: "Formatted at 2025-01-09 20:58:41 on dist-test-slave-j54z"
I20250109 20:58:41.705575 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250109 20:58:41.708875 19615 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:41.709573 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:58:41.709828 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "332a708cffb544b9b79e8b7495614180"
format_stamp: "Formatted at 2025-01-09 20:58:41 on dist-test-slave-j54z"
I20250109 20:58:41.710099 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:41.723357 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:41.724525 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:41.760418 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:43793
I20250109 20:58:41.760504 19666 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:43793 every 8 connection(s)
I20250109 20:58:41.764391 19667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:41.774366 19667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180: Bootstrap starting.
I20250109 20:58:41.778724 19667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:41.782755 19667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180: No bootstrap required, opened a new log
I20250109 20:58:41.784751 19667 raft_consensus.cc:357] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332a708cffb544b9b79e8b7495614180" member_type: VOTER }
I20250109 20:58:41.785230 19667 raft_consensus.cc:383] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:41.785454 19667 raft_consensus.cc:738] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332a708cffb544b9b79e8b7495614180, State: Initialized, Role: FOLLOWER
I20250109 20:58:41.786065 19667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332a708cffb544b9b79e8b7495614180" member_type: VOTER }
I20250109 20:58:41.786494 19667 raft_consensus.cc:397] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:58:41.786700 19667 raft_consensus.cc:491] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:58:41.786964 19667 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:41.791074 19667 raft_consensus.cc:513] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332a708cffb544b9b79e8b7495614180" member_type: VOTER }
I20250109 20:58:41.791517 19667 leader_election.cc:304] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 332a708cffb544b9b79e8b7495614180; no voters:
I20250109 20:58:41.792538 19667 leader_election.cc:290] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:58:41.792909 19670 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:41.794185 19670 raft_consensus.cc:695] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [term 1 LEADER]: Becoming Leader. State: Replica: 332a708cffb544b9b79e8b7495614180, State: Running, Role: LEADER
I20250109 20:58:41.794742 19670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332a708cffb544b9b79e8b7495614180" member_type: VOTER }
I20250109 20:58:41.795450 19667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:58:41.799286 19671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "332a708cffb544b9b79e8b7495614180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332a708cffb544b9b79e8b7495614180" member_type: VOTER } }
I20250109 20:58:41.799893 19671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [sys.catalog]: This master's current role is: LEADER
I20250109 20:58:41.800199 19672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 332a708cffb544b9b79e8b7495614180. Latest consensus state: current_term: 1 leader_uuid: "332a708cffb544b9b79e8b7495614180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332a708cffb544b9b79e8b7495614180" member_type: VOTER } }
I20250109 20:58:41.800943 19672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [sys.catalog]: This master's current role is: LEADER
I20250109 20:58:41.803443 19678 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:58:41.811389 19678 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:58:41.812466 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 20:58:41.819720 19678 catalog_manager.cc:1348] Generated new cluster ID: aebe953d37654e93afeb2ceb72aa82d7
I20250109 20:58:41.820003 19678 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:58:41.838372 19678 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 20:58:41.839561 19678 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:58:41.852937 19678 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180: Generated new TSK 0
I20250109 20:58:41.853495 19678 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:58:41.878057 17466 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.000217133s
I20250109 20:58:53.972411 17466 master.cc:537] Master@127.17.14.190:43793 shutting down...
I20250109 20:58:53.995776 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:58:53.996520 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:58:53.996857 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P 332a708cffb544b9b79e8b7495614180: stopping tablet replica
I20250109 20:58:54.015887 17466 master.cc:559] Master@127.17.14.190:43793 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12360 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20250109 20:58:54.046939 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:41865
I20250109 20:58:54.048015 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:54.054060 19690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:54.055322 19691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:54.056385 19693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:54.057090 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:54.057794 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:54.057971 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:54.058082 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456334058072 us; error 0 us; skew 500 ppm
I20250109 20:58:54.058487 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:54.060784 17466 webserver.cc:458] Webserver started at http://127.17.14.190:38789/ using document root <none> and password file <none>
I20250109 20:58:54.061228 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:54.061375 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:54.061635 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:54.062814 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "574f4c3cea934dbcb340224dbb74fa4f"
format_stamp: "Formatted at 2025-01-09 20:58:54 on dist-test-slave-j54z"
I20250109 20:58:54.067341 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250109 20:58:54.070566 19698 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:54.071295 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:58:54.071556 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "574f4c3cea934dbcb340224dbb74fa4f"
format_stamp: "Formatted at 2025-01-09 20:58:54 on dist-test-slave-j54z"
I20250109 20:58:54.071813 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:54.081322 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:54.082533 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:54.123776 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:41865
I20250109 20:58:54.123872 19749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:41865 every 8 connection(s)
I20250109 20:58:54.127641 19750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:58:54.138801 19750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f: Bootstrap starting.
I20250109 20:58:54.143236 19750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f: Neither blocks nor log segments found. Creating new log.
I20250109 20:58:54.147300 19750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f: No bootstrap required, opened a new log
I20250109 20:58:54.149365 19750 raft_consensus.cc:357] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574f4c3cea934dbcb340224dbb74fa4f" member_type: VOTER }
I20250109 20:58:54.149886 19750 raft_consensus.cc:383] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:58:54.150104 19750 raft_consensus.cc:738] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574f4c3cea934dbcb340224dbb74fa4f, State: Initialized, Role: FOLLOWER
I20250109 20:58:54.150624 19750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574f4c3cea934dbcb340224dbb74fa4f" member_type: VOTER }
I20250109 20:58:54.151064 19750 raft_consensus.cc:397] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:58:54.151278 19750 raft_consensus.cc:491] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:58:54.151551 19750 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:58:54.155958 19750 raft_consensus.cc:513] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574f4c3cea934dbcb340224dbb74fa4f" member_type: VOTER }
I20250109 20:58:54.156452 19750 leader_election.cc:304] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 574f4c3cea934dbcb340224dbb74fa4f; no voters:
I20250109 20:58:54.157558 19750 leader_election.cc:290] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:58:54.157984 19753 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:58:54.159227 19753 raft_consensus.cc:695] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [term 1 LEADER]: Becoming Leader. State: Replica: 574f4c3cea934dbcb340224dbb74fa4f, State: Running, Role: LEADER
I20250109 20:58:54.159900 19753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574f4c3cea934dbcb340224dbb74fa4f" member_type: VOTER }
I20250109 20:58:54.160465 19750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:58:54.162854 19754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "574f4c3cea934dbcb340224dbb74fa4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574f4c3cea934dbcb340224dbb74fa4f" member_type: VOTER } }
I20250109 20:58:54.163602 19754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [sys.catalog]: This master's current role is: LEADER
I20250109 20:58:54.162865 19755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 574f4c3cea934dbcb340224dbb74fa4f. Latest consensus state: current_term: 1 leader_uuid: "574f4c3cea934dbcb340224dbb74fa4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574f4c3cea934dbcb340224dbb74fa4f" member_type: VOTER } }
I20250109 20:58:54.164301 19755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [sys.catalog]: This master's current role is: LEADER
I20250109 20:58:54.166810 19758 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:58:54.172456 19758 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:58:54.175014 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 20:58:54.181448 19758 catalog_manager.cc:1348] Generated new cluster ID: d0ab5733a04a490c9df41908116f9579
I20250109 20:58:54.181721 19758 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:58:54.203698 19758 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 20:58:54.204911 19758 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:58:54.217046 19758 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f: Generated new TSK 0
I20250109 20:58:54.217615 19758 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:58:54.241291 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:54.247716 19771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:54.248816 19772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:54.249529 19774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:54.250479 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:54.251238 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:54.251423 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:54.251566 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456334251553 us; error 0 us; skew 500 ppm
I20250109 20:58:54.252048 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:54.254429 17466 webserver.cc:458] Webserver started at http://127.17.14.129:41575/ using document root <none> and password file <none>
I20250109 20:58:54.254851 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:54.255012 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:54.255261 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:54.256392 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-0-root/instance:
uuid: "6790e30b6df74a5eb229b069db4df8d3"
format_stamp: "Formatted at 2025-01-09 20:58:54 on dist-test-slave-j54z"
I20250109 20:58:54.260798 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250109 20:58:54.264084 19779 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:54.264827 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 20:58:54.265092 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "6790e30b6df74a5eb229b069db4df8d3"
format_stamp: "Formatted at 2025-01-09 20:58:54 on dist-test-slave-j54z"
I20250109 20:58:54.265362 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:54.277236 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:54.278374 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:54.279760 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:58:54.281977 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:58:54.282161 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:54.282397 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:58:54.282541 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:54.321807 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:41207
I20250109 20:58:54.321882 19841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:41207 every 8 connection(s)
I20250109 20:58:54.326478 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:54.334975 19846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:54.336333 19847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:54.338774 19849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:54.340373 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:54.340706 19842 heartbeater.cc:346] Connected to a master server at 127.17.14.190:41865
I20250109 20:58:54.341049 19842 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:54.341207 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:54.341445 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:54.341655 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456334341637 us; error 0 us; skew 500 ppm
I20250109 20:58:54.341826 19842 heartbeater.cc:510] Master 127.17.14.190:41865 requested a full tablet report, sending...
I20250109 20:58:54.342200 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:54.344069 19715 ts_manager.cc:194] Registered new tserver with Master: 6790e30b6df74a5eb229b069db4df8d3 (127.17.14.129:41207)
I20250109 20:58:54.345026 17466 webserver.cc:458] Webserver started at http://127.17.14.130:43793/ using document root <none> and password file <none>
I20250109 20:58:54.345521 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:54.345701 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:54.345949 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:54.345980 19715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43230
I20250109 20:58:54.347339 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-1-root/instance:
uuid: "f6059b461ae2485f94f225f346699eed"
format_stamp: "Formatted at 2025-01-09 20:58:54 on dist-test-slave-j54z"
I20250109 20:58:54.351574 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250109 20:58:54.354571 19854 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:54.355232 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 20:58:54.355485 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "f6059b461ae2485f94f225f346699eed"
format_stamp: "Formatted at 2025-01-09 20:58:54 on dist-test-slave-j54z"
I20250109 20:58:54.355736 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:54.369402 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:54.370625 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:54.372006 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:58:54.374433 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:58:54.374627 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:54.374843 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:58:54.375000 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:54.413317 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:43021
I20250109 20:58:54.413407 19916 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:43021 every 8 connection(s)
I20250109 20:58:54.417897 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:58:54.425770 19920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:54.427150 19921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:58:54.428625 19923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:58:54.429221 19917 heartbeater.cc:346] Connected to a master server at 127.17.14.190:41865
I20250109 20:58:54.429555 19917 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:54.430159 17466 server_base.cc:1034] running on GCE node
I20250109 20:58:54.430290 19917 heartbeater.cc:510] Master 127.17.14.190:41865 requested a full tablet report, sending...
I20250109 20:58:54.430994 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:58:54.431267 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:58:54.431473 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456334431458 us; error 0 us; skew 500 ppm
I20250109 20:58:54.431946 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:58:54.432286 19715 ts_manager.cc:194] Registered new tserver with Master: f6059b461ae2485f94f225f346699eed (127.17.14.130:43021)
I20250109 20:58:54.433758 19715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43238
I20250109 20:58:54.434746 17466 webserver.cc:458] Webserver started at http://127.17.14.131:45681/ using document root <none> and password file <none>
I20250109 20:58:54.435261 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:58:54.435424 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:58:54.435667 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:58:54.436735 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-2-root/instance:
uuid: "523f445900f846baa5d47b15349d7ea0"
format_stamp: "Formatted at 2025-01-09 20:58:54 on dist-test-slave-j54z"
I20250109 20:58:54.440976 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 20:58:54.444070 19928 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:54.444758 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 20:58:54.445037 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "523f445900f846baa5d47b15349d7ea0"
format_stamp: "Formatted at 2025-01-09 20:58:54 on dist-test-slave-j54z"
I20250109 20:58:54.445281 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:58:54.459257 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:58:54.460364 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:58:54.461719 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:58:54.463862 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:58:54.464041 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:54.464268 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:58:54.464403 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:58:54.502831 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:41859
I20250109 20:58:54.502940 19990 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:41859 every 8 connection(s)
I20250109 20:58:54.515473 19991 heartbeater.cc:346] Connected to a master server at 127.17.14.190:41865
I20250109 20:58:54.515836 19991 heartbeater.cc:463] Registering TS with master...
I20250109 20:58:54.516564 19991 heartbeater.cc:510] Master 127.17.14.190:41865 requested a full tablet report, sending...
I20250109 20:58:54.518275 19715 ts_manager.cc:194] Registered new tserver with Master: 523f445900f846baa5d47b15349d7ea0 (127.17.14.131:41859)
I20250109 20:58:54.519201 17466 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013324134s
I20250109 20:58:54.519551 19715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43244
I20250109 20:58:55.348448 19842 heartbeater.cc:502] Master 127.17.14.190:41865 was elected leader, sending a full tablet report...
I20250109 20:58:55.436235 19917 heartbeater.cc:502] Master 127.17.14.190:41865 was elected leader, sending a full tablet report...
I20250109 20:58:55.521821 19991 heartbeater.cc:502] Master 127.17.14.190:41865 was elected leader, sending a full tablet report...
I20250109 20:58:56.174809 19768 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:58:58.175695 19768 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:00.176626 19768 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:02.177709 19768 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:04.178822 19768 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:06.179917 19768 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:06.613302 17466 tablet_server.cc:178] TabletServer@127.17.14.129:0 shutting down...
I20250109 20:59:06.637782 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:06.654556 17466 tablet_server.cc:195] TabletServer@127.17.14.129:0 shutdown complete.
I20250109 20:59:06.663844 17466 tablet_server.cc:178] TabletServer@127.17.14.130:0 shutting down...
I20250109 20:59:06.681639 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:06.697263 17466 tablet_server.cc:195] TabletServer@127.17.14.130:0 shutdown complete.
I20250109 20:59:06.706005 17466 tablet_server.cc:178] TabletServer@127.17.14.131:0 shutting down...
I20250109 20:59:06.723881 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:06.739741 17466 tablet_server.cc:195] TabletServer@127.17.14.131:0 shutdown complete.
I20250109 20:59:06.747865 17466 master.cc:537] Master@127.17.14.190:41865 shutting down...
I20250109 20:59:06.764215 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:59:06.764940 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:06.765251 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P 574f4c3cea934dbcb340224dbb74fa4f: stopping tablet replica
I20250109 20:59:06.783047 17466 master.cc:559] Master@127.17.14.190:41865 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12764 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20250109 20:59:06.811103 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:42199
I20250109 20:59:06.812193 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:06.818329 19996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:06.818969 19997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:06.820019 19999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:06.820652 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:06.821457 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:06.821692 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:06.821841 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456346821820 us; error 0 us; skew 500 ppm
I20250109 20:59:06.822322 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:06.830812 17466 webserver.cc:458] Webserver started at http://127.17.14.190:39059/ using document root <none> and password file <none>
I20250109 20:59:06.831293 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:06.831461 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:06.831718 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:06.832998 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "b5dc125bd5724e3c85f41de0aa07cd6e"
format_stamp: "Formatted at 2025-01-09 20:59:06 on dist-test-slave-j54z"
I20250109 20:59:06.837514 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20250109 20:59:06.840765 20004 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:06.841457 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 20:59:06.841738 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "b5dc125bd5724e3c85f41de0aa07cd6e"
format_stamp: "Formatted at 2025-01-09 20:59:06 on dist-test-slave-j54z"
I20250109 20:59:06.842003 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:06.860281 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:06.861541 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:06.897289 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:42199
I20250109 20:59:06.897379 20055 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:42199 every 8 connection(s)
I20250109 20:59:06.901153 20056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:06.911680 20056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e: Bootstrap starting.
I20250109 20:59:06.916075 20056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:06.920333 20056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e: No bootstrap required, opened a new log
I20250109 20:59:06.922377 20056 raft_consensus.cc:357] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dc125bd5724e3c85f41de0aa07cd6e" member_type: VOTER }
I20250109 20:59:06.922828 20056 raft_consensus.cc:383] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:06.922997 20056 raft_consensus.cc:738] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5dc125bd5724e3c85f41de0aa07cd6e, State: Initialized, Role: FOLLOWER
I20250109 20:59:06.923520 20056 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dc125bd5724e3c85f41de0aa07cd6e" member_type: VOTER }
I20250109 20:59:06.923933 20056 raft_consensus.cc:397] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:59:06.924108 20056 raft_consensus.cc:491] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:59:06.924343 20056 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:06.928491 20056 raft_consensus.cc:513] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dc125bd5724e3c85f41de0aa07cd6e" member_type: VOTER }
I20250109 20:59:06.928980 20056 leader_election.cc:304] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5dc125bd5724e3c85f41de0aa07cd6e; no voters:
I20250109 20:59:06.930038 20056 leader_election.cc:290] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:59:06.930426 20059 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:59:06.931648 20059 raft_consensus.cc:695] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [term 1 LEADER]: Becoming Leader. State: Replica: b5dc125bd5724e3c85f41de0aa07cd6e, State: Running, Role: LEADER
I20250109 20:59:06.932319 20059 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dc125bd5724e3c85f41de0aa07cd6e" member_type: VOTER }
I20250109 20:59:06.932886 20056 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:59:06.935221 20061 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5dc125bd5724e3c85f41de0aa07cd6e. Latest consensus state: current_term: 1 leader_uuid: "b5dc125bd5724e3c85f41de0aa07cd6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dc125bd5724e3c85f41de0aa07cd6e" member_type: VOTER } }
I20250109 20:59:06.935199 20060 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5dc125bd5724e3c85f41de0aa07cd6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dc125bd5724e3c85f41de0aa07cd6e" member_type: VOTER } }
I20250109 20:59:06.935900 20061 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [sys.catalog]: This master's current role is: LEADER
I20250109 20:59:06.936021 20060 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [sys.catalog]: This master's current role is: LEADER
I20250109 20:59:06.940140 20066 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:59:06.948297 20066 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:59:06.949380 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 20:59:06.956619 20066 catalog_manager.cc:1348] Generated new cluster ID: b362ca1c92084c95a9565a703f12e779
I20250109 20:59:06.956861 20066 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:59:06.971805 20066 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 20:59:06.972973 20066 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:59:06.985230 20066 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e: Generated new TSK 0
I20250109 20:59:06.985742 20066 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:59:07.015941 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:07.022212 20077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:07.023628 20078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:07.024243 20080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:07.025688 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:07.026456 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:07.026634 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:07.026778 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456347026763 us; error 0 us; skew 500 ppm
I20250109 20:59:07.027241 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:07.029510 17466 webserver.cc:458] Webserver started at http://127.17.14.129:46285/ using document root <none> and password file <none>
I20250109 20:59:07.029938 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:07.030107 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:07.030354 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:07.031431 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-0-root/instance:
uuid: "ede440e9a7114f7aa15745281b69f7a3"
format_stamp: "Formatted at 2025-01-09 20:59:07 on dist-test-slave-j54z"
I20250109 20:59:07.035925 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 20:59:07.039053 20085 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:07.039757 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 20:59:07.040009 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "ede440e9a7114f7aa15745281b69f7a3"
format_stamp: "Formatted at 2025-01-09 20:59:07 on dist-test-slave-j54z"
I20250109 20:59:07.040277 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:07.059863 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:07.061055 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:07.062534 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:07.064754 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:07.064946 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:07.065169 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:07.065316 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:07.103060 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:42123
I20250109 20:59:07.103148 20147 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:42123 every 8 connection(s)
I20250109 20:59:07.107529 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:07.115921 20152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:07.118002 20153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:07.119139 20155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:07.120330 20148 heartbeater.cc:346] Connected to a master server at 127.17.14.190:42199
I20250109 20:59:07.120666 20148 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:07.121117 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:07.121419 20148 heartbeater.cc:510] Master 127.17.14.190:42199 requested a full tablet report, sending...
I20250109 20:59:07.122115 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:07.122361 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:07.122541 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456347122520 us; error 0 us; skew 500 ppm
I20250109 20:59:07.123096 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:07.123790 20021 ts_manager.cc:194] Registered new tserver with Master: ede440e9a7114f7aa15745281b69f7a3 (127.17.14.129:42123)
I20250109 20:59:07.125952 17466 webserver.cc:458] Webserver started at http://127.17.14.130:44811/ using document root <none> and password file <none>
I20250109 20:59:07.126364 20021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39228
I20250109 20:59:07.126564 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:07.126824 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:07.127167 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:07.128744 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-1-root/instance:
uuid: "d3b9dadddd054703976ba5e86f09ef37"
format_stamp: "Formatted at 2025-01-09 20:59:07 on dist-test-slave-j54z"
I20250109 20:59:07.133013 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250109 20:59:07.136171 20160 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:07.136848 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 20:59:07.137094 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "d3b9dadddd054703976ba5e86f09ef37"
format_stamp: "Formatted at 2025-01-09 20:59:07 on dist-test-slave-j54z"
I20250109 20:59:07.137358 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:07.152595 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:07.153767 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:07.155141 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:07.157275 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:07.157498 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:07.157721 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:07.157869 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:07.195647 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:38865
I20250109 20:59:07.195744 20222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:38865 every 8 connection(s)
I20250109 20:59:07.200233 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:07.207492 20226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:07.208402 20227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:07.210197 20229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:07.211319 20223 heartbeater.cc:346] Connected to a master server at 127.17.14.190:42199
I20250109 20:59:07.211699 20223 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:07.211999 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:07.212484 20223 heartbeater.cc:510] Master 127.17.14.190:42199 requested a full tablet report, sending...
I20250109 20:59:07.212852 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:07.213070 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:07.213238 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456347213218 us; error 0 us; skew 500 ppm
I20250109 20:59:07.213922 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:07.214460 20021 ts_manager.cc:194] Registered new tserver with Master: d3b9dadddd054703976ba5e86f09ef37 (127.17.14.130:38865)
I20250109 20:59:07.216111 20021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39236
I20250109 20:59:07.216792 17466 webserver.cc:458] Webserver started at http://127.17.14.131:40705/ using document root <none> and password file <none>
I20250109 20:59:07.217379 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:07.217598 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:07.217846 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:07.219035 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-2-root/instance:
uuid: "24d1d60a52a5406db7826d7a52c719e5"
format_stamp: "Formatted at 2025-01-09 20:59:07 on dist-test-slave-j54z"
I20250109 20:59:07.223421 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250109 20:59:07.226562 20234 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:07.227262 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 20:59:07.227530 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "24d1d60a52a5406db7826d7a52c719e5"
format_stamp: "Formatted at 2025-01-09 20:59:07 on dist-test-slave-j54z"
I20250109 20:59:07.227794 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:07.237591 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:07.238693 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:07.240033 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:07.242285 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:07.242497 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:07.242758 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:07.242909 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:07.280470 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:41773
I20250109 20:59:07.280563 20296 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:41773 every 8 connection(s)
I20250109 20:59:07.284991 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:07.292896 20300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:07.294651 20301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:07.296222 20303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:07.297722 20297 heartbeater.cc:346] Connected to a master server at 127.17.14.190:42199
I20250109 20:59:07.298014 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:07.298178 20297 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:07.299047 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250109 20:59:07.299140 20297 heartbeater.cc:510] Master 127.17.14.190:42199 requested a full tablet report, sending...
W20250109 20:59:07.299358 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:07.299599 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456347299579 us; error 0 us; skew 500 ppm
I20250109 20:59:07.300222 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:07.301393 20021 ts_manager.cc:194] Registered new tserver with Master: 24d1d60a52a5406db7826d7a52c719e5 (127.17.14.131:41773)
I20250109 20:59:07.303032 17466 webserver.cc:458] Webserver started at http://127.17.14.132:37871/ using document root <none> and password file <none>
I20250109 20:59:07.303282 20021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39238
I20250109 20:59:07.303661 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:07.303925 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:07.304222 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:07.305792 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-3-root/instance:
uuid: "0f60d1a295e14b4e8af709322f6dadbb"
format_stamp: "Formatted at 2025-01-09 20:59:07 on dist-test-slave-j54z"
I20250109 20:59:07.310110 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250109 20:59:07.313274 20308 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:07.314030 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 20:59:07.314312 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-3-root
uuid: "0f60d1a295e14b4e8af709322f6dadbb"
format_stamp: "Formatted at 2025-01-09 20:59:07 on dist-test-slave-j54z"
I20250109 20:59:07.314592 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736456270445890-17466-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:07.340490 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:07.342012 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:07.343674 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:07.345814 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:07.346009 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:07.346215 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:07.346366 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:07.384338 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.132:45589
I20250109 20:59:07.384425 20370 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.132:45589 every 8 connection(s)
I20250109 20:59:07.396997 20371 heartbeater.cc:346] Connected to a master server at 127.17.14.190:42199
I20250109 20:59:07.397361 20371 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:07.398088 20371 heartbeater.cc:510] Master 127.17.14.190:42199 requested a full tablet report, sending...
I20250109 20:59:07.399788 20021 ts_manager.cc:194] Registered new tserver with Master: 0f60d1a295e14b4e8af709322f6dadbb (127.17.14.132:45589)
I20250109 20:59:07.400558 17466 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.013304084s
I20250109 20:59:07.401060 20021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39244
I20250109 20:59:08.129405 20148 heartbeater.cc:502] Master 127.17.14.190:42199 was elected leader, sending a full tablet report...
I20250109 20:59:08.218688 20223 heartbeater.cc:502] Master 127.17.14.190:42199 was elected leader, sending a full tablet report...
I20250109 20:59:08.306077 20297 heartbeater.cc:502] Master 127.17.14.190:42199 was elected leader, sending a full tablet report...
I20250109 20:59:08.403784 20371 heartbeater.cc:502] Master 127.17.14.190:42199 was elected leader, sending a full tablet report...
I20250109 20:59:08.444049 17466 test_util.cc:274] Using random seed: 1070671228
I20250109 20:59:08.463713 20021 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39258:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:59:08.501897 20262 tablet_service.cc:1467] Processing CreateTablet for tablet e84a6233b1d6435488e2baf8bbb34462 (DEFAULT_TABLE table=test-workload [id=b027022c697d41f288103b2ab0f279aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:59:08.502866 20262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e84a6233b1d6435488e2baf8bbb34462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:08.506767 20113 tablet_service.cc:1467] Processing CreateTablet for tablet e84a6233b1d6435488e2baf8bbb34462 (DEFAULT_TABLE table=test-workload [id=b027022c697d41f288103b2ab0f279aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:59:08.507774 20113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e84a6233b1d6435488e2baf8bbb34462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:08.511929 20188 tablet_service.cc:1467] Processing CreateTablet for tablet e84a6233b1d6435488e2baf8bbb34462 (DEFAULT_TABLE table=test-workload [id=b027022c697d41f288103b2ab0f279aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:59:08.513147 20188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e84a6233b1d6435488e2baf8bbb34462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:08.520354 20392 tablet_bootstrap.cc:492] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5: Bootstrap starting.
I20250109 20:59:08.522923 20393 tablet_bootstrap.cc:492] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3: Bootstrap starting.
I20250109 20:59:08.526347 20392 tablet_bootstrap.cc:654] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:08.529979 20393 tablet_bootstrap.cc:654] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:08.533537 20394 tablet_bootstrap.cc:492] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37: Bootstrap starting.
I20250109 20:59:08.534387 20392 tablet_bootstrap.cc:492] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5: No bootstrap required, opened a new log
I20250109 20:59:08.534880 20392 ts_tablet_manager.cc:1397] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250109 20:59:08.535749 20393 tablet_bootstrap.cc:492] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3: No bootstrap required, opened a new log
I20250109 20:59:08.536103 20393 ts_tablet_manager.cc:1397] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20250109 20:59:08.537781 20392 raft_consensus.cc:357] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d1d60a52a5406db7826d7a52c719e5" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 41773 } } peers { permanent_uuid: "ede440e9a7114f7aa15745281b69f7a3" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 42123 } } peers { permanent_uuid: "d3b9dadddd054703976ba5e86f09ef37" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38865 } }
I20250109 20:59:08.538219 20393 raft_consensus.cc:357] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d1d60a52a5406db7826d7a52c719e5" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 41773 } } peers { permanent_uuid: "ede440e9a7114f7aa15745281b69f7a3" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 42123 } } peers { permanent_uuid: "d3b9dadddd054703976ba5e86f09ef37" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38865 } }
I20250109 20:59:08.538704 20392 raft_consensus.cc:383] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:08.538882 20393 raft_consensus.cc:383] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:08.539157 20392 raft_consensus.cc:738] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24d1d60a52a5406db7826d7a52c719e5, State: Initialized, Role: FOLLOWER
I20250109 20:59:08.539268 20393 raft_consensus.cc:738] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede440e9a7114f7aa15745281b69f7a3, State: Initialized, Role: FOLLOWER
I20250109 20:59:08.540179 20393 consensus_queue.cc:260] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d1d60a52a5406db7826d7a52c719e5" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 41773 } } peers { permanent_uuid: "ede440e9a7114f7aa15745281b69f7a3" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 42123 } } peers { permanent_uuid: "d3b9dadddd054703976ba5e86f09ef37" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38865 } }
I20250109 20:59:08.540529 20394 tablet_bootstrap.cc:654] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:08.540529 20392 consensus_queue.cc:260] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d1d60a52a5406db7826d7a52c719e5" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 41773 } } peers { permanent_uuid: "ede440e9a7114f7aa15745281b69f7a3" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 42123 } } peers { permanent_uuid: "d3b9dadddd054703976ba5e86f09ef37" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38865 } }
I20250109 20:59:08.543432 20393 ts_tablet_manager.cc:1428] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20250109 20:59:08.545840 20392 ts_tablet_manager.cc:1428] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20250109 20:59:08.546598 20394 tablet_bootstrap.cc:492] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37: No bootstrap required, opened a new log
I20250109 20:59:08.547103 20394 ts_tablet_manager.cc:1397] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20250109 20:59:08.550076 20394 raft_consensus.cc:357] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d1d60a52a5406db7826d7a52c719e5" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 41773 } } peers { permanent_uuid: "ede440e9a7114f7aa15745281b69f7a3" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 42123 } } peers { permanent_uuid: "d3b9dadddd054703976ba5e86f09ef37" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38865 } }
I20250109 20:59:08.550825 20394 raft_consensus.cc:383] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:08.551151 20394 raft_consensus.cc:738] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b9dadddd054703976ba5e86f09ef37, State: Initialized, Role: FOLLOWER
I20250109 20:59:08.552031 20394 consensus_queue.cc:260] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d1d60a52a5406db7826d7a52c719e5" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 41773 } } peers { permanent_uuid: "ede440e9a7114f7aa15745281b69f7a3" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 42123 } } peers { permanent_uuid: "d3b9dadddd054703976ba5e86f09ef37" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38865 } }
I20250109 20:59:08.557252 20394 ts_tablet_manager.cc:1428] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20250109 20:59:08.593561 20400 raft_consensus.cc:491] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:59:08.594014 20400 raft_consensus.cc:513] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d1d60a52a5406db7826d7a52c719e5" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 41773 } } peers { permanent_uuid: "ede440e9a7114f7aa15745281b69f7a3" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 42123 } } peers { permanent_uuid: "d3b9dadddd054703976ba5e86f09ef37" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38865 } }
I20250109 20:59:08.595913 20400 leader_election.cc:290] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24d1d60a52a5406db7826d7a52c719e5 (127.17.14.131:41773), ede440e9a7114f7aa15745281b69f7a3 (127.17.14.129:42123)
I20250109 20:59:08.607349 20123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a6233b1d6435488e2baf8bbb34462" candidate_uuid: "d3b9dadddd054703976ba5e86f09ef37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede440e9a7114f7aa15745281b69f7a3" is_pre_election: true
I20250109 20:59:08.608103 20123 raft_consensus.cc:2463] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3b9dadddd054703976ba5e86f09ef37 in term 0.
I20250109 20:59:08.608042 20272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a6233b1d6435488e2baf8bbb34462" candidate_uuid: "d3b9dadddd054703976ba5e86f09ef37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24d1d60a52a5406db7826d7a52c719e5" is_pre_election: true
W20250109 20:59:08.608492 20149 tablet.cc:2367] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 20:59:08.608850 20272 raft_consensus.cc:2463] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3b9dadddd054703976ba5e86f09ef37 in term 0.
I20250109 20:59:08.609325 20161 leader_election.cc:304] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3b9dadddd054703976ba5e86f09ef37, ede440e9a7114f7aa15745281b69f7a3; no voters:
I20250109 20:59:08.610010 20400 raft_consensus.cc:2798] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:59:08.610265 20400 raft_consensus.cc:491] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:59:08.610524 20400 raft_consensus.cc:3054] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:08.615228 20400 raft_consensus.cc:513] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d1d60a52a5406db7826d7a52c719e5" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 41773 } } peers { permanent_uuid: "ede440e9a7114f7aa15745281b69f7a3" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 42123 } } peers { permanent_uuid: "d3b9dadddd054703976ba5e86f09ef37" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38865 } }
I20250109 20:59:08.616669 20400 leader_election.cc:290] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [CANDIDATE]: Term 1 election: Requested vote from peers 24d1d60a52a5406db7826d7a52c719e5 (127.17.14.131:41773), ede440e9a7114f7aa15745281b69f7a3 (127.17.14.129:42123)
I20250109 20:59:08.617445 20272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a6233b1d6435488e2baf8bbb34462" candidate_uuid: "d3b9dadddd054703976ba5e86f09ef37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24d1d60a52a5406db7826d7a52c719e5"
I20250109 20:59:08.617605 20123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a6233b1d6435488e2baf8bbb34462" candidate_uuid: "d3b9dadddd054703976ba5e86f09ef37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede440e9a7114f7aa15745281b69f7a3"
I20250109 20:59:08.617945 20272 raft_consensus.cc:3054] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:08.618098 20123 raft_consensus.cc:3054] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:08.623049 20123 raft_consensus.cc:2463] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3b9dadddd054703976ba5e86f09ef37 in term 1.
I20250109 20:59:08.623114 20272 raft_consensus.cc:2463] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3b9dadddd054703976ba5e86f09ef37 in term 1.
I20250109 20:59:08.623980 20161 leader_election.cc:304] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3b9dadddd054703976ba5e86f09ef37, ede440e9a7114f7aa15745281b69f7a3; no voters:
I20250109 20:59:08.624555 20400 raft_consensus.cc:2798] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:59:08.625609 20400 raft_consensus.cc:695] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 1 LEADER]: Becoming Leader. State: Replica: d3b9dadddd054703976ba5e86f09ef37, State: Running, Role: LEADER
I20250109 20:59:08.626225 20400 consensus_queue.cc:237] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d1d60a52a5406db7826d7a52c719e5" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 41773 } } peers { permanent_uuid: "ede440e9a7114f7aa15745281b69f7a3" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 42123 } } peers { permanent_uuid: "d3b9dadddd054703976ba5e86f09ef37" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38865 } }
I20250109 20:59:08.634153 20021 catalog_manager.cc:5526] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3b9dadddd054703976ba5e86f09ef37 (127.17.14.130). New cstate: current_term: 1 leader_uuid: "d3b9dadddd054703976ba5e86f09ef37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d1d60a52a5406db7826d7a52c719e5" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 41773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede440e9a7114f7aa15745281b69f7a3" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3b9dadddd054703976ba5e86f09ef37" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38865 } health_report { overall_health: HEALTHY } } }
I20250109 20:59:08.947779 20074 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:08.949323 20074 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:08.949667 20074 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:09.146019 20405 consensus_queue.cc:1035] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede440e9a7114f7aa15745281b69f7a3" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:59:09.146662 20406 consensus_queue.cc:1035] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24d1d60a52a5406db7826d7a52c719e5" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 41773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:59:10.950582 20074 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:10.951826 20074 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:10.952096 20074 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250109 20:59:11.240890 20219 debug-util.cc:398] Leaking SignalData structure 0x7b08001d27e0 after lost signal to thread 17469
W20250109 20:59:11.242069 20219 debug-util.cc:398] Leaking SignalData structure 0x7b0800061560 after lost signal to thread 20055
W20250109 20:59:11.243108 20219 debug-util.cc:398] Leaking SignalData structure 0x7b08000a6240 after lost signal to thread 20147
W20250109 20:59:11.244055 20219 debug-util.cc:398] Leaking SignalData structure 0x7b08001a86a0 after lost signal to thread 20222
W20250109 20:59:11.245010 20219 debug-util.cc:398] Leaking SignalData structure 0x7b080005af20 after lost signal to thread 20296
W20250109 20:59:11.246014 20219 debug-util.cc:398] Leaking SignalData structure 0x7b080005cfe0 after lost signal to thread 20370
I20250109 20:59:12.953158 20074 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:12.954574 20074 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:12.954875 20074 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:15.850098 20074 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:15.851990 20074 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:15.859088 20074 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:15.857823 20418 raft_consensus.cc:491] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3b9dadddd054703976ba5e86f09ef37)
I20250109 20:59:15.859784 20418 raft_consensus.cc:513] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24d1d60a52a5406db7826d7a52c719e5" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 41773 } } peers { permanent_uuid: "ede440e9a7114f7aa15745281b69f7a3" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 42123 } } peers { permanent_uuid: "d3b9dadddd054703976ba5e86f09ef37" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38865 } }
I20250109 20:59:15.864373 20418 leader_election.cc:290] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24d1d60a52a5406db7826d7a52c719e5 (127.17.14.131:41773), d3b9dadddd054703976ba5e86f09ef37 (127.17.14.130:38865)
I20250109 20:59:15.987202 20272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a6233b1d6435488e2baf8bbb34462" candidate_uuid: "ede440e9a7114f7aa15745281b69f7a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "24d1d60a52a5406db7826d7a52c719e5" is_pre_election: true
I20250109 20:59:15.995929 20198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a6233b1d6435488e2baf8bbb34462" candidate_uuid: "ede440e9a7114f7aa15745281b69f7a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3b9dadddd054703976ba5e86f09ef37" is_pre_election: true
I20250109 20:59:15.997186 20086 leader_election.cc:304] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ede440e9a7114f7aa15745281b69f7a3; no voters: 24d1d60a52a5406db7826d7a52c719e5, d3b9dadddd054703976ba5e86f09ef37
I20250109 20:59:15.997925 20418 raft_consensus.cc:2743] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 20:59:16.939018 20367 debug-util.cc:398] Leaking SignalData structure 0x7b0800159fc0 after lost signal to thread 17469
W20250109 20:59:16.939860 20367 debug-util.cc:398] Leaking SignalData structure 0x7b0800156480 after lost signal to thread 20055
W20250109 20:59:16.940845 20367 debug-util.cc:398] Leaking SignalData structure 0x7b08001219e0 after lost signal to thread 20147
W20250109 20:59:16.941679 20367 debug-util.cc:398] Leaking SignalData structure 0x7b08000a6100 after lost signal to thread 20222
W20250109 20:59:16.942525 20367 debug-util.cc:398] Leaking SignalData structure 0x7b08000355a0 after lost signal to thread 20296
W20250109 20:59:16.943267 20367 debug-util.cc:398] Leaking SignalData structure 0x7b08001b17e0 after lost signal to thread 20370
I20250109 20:59:17.860178 20074 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:17.861783 20074 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:17.862068 20074 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:19.863009 20074 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:19.864274 20074 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:19.864559 20074 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:20.776434 17466 tablet_server.cc:178] TabletServer@127.17.14.129:0 shutting down...
I20250109 20:59:20.795426 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:20.796077 17466 tablet_replica.cc:331] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3: stopping tablet replica
I20250109 20:59:20.796630 17466 raft_consensus.cc:2238] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:59:20.797088 17466 raft_consensus.cc:2267] T e84a6233b1d6435488e2baf8bbb34462 P ede440e9a7114f7aa15745281b69f7a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:20.816651 17466 tablet_server.cc:195] TabletServer@127.17.14.129:0 shutdown complete.
I20250109 20:59:20.827068 17466 tablet_server.cc:178] TabletServer@127.17.14.130:0 shutting down...
I20250109 20:59:20.844609 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:20.845257 17466 tablet_replica.cc:331] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37: stopping tablet replica
I20250109 20:59:20.845834 17466 raft_consensus.cc:2238] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:59:20.846596 17466 raft_consensus.cc:2267] T e84a6233b1d6435488e2baf8bbb34462 P d3b9dadddd054703976ba5e86f09ef37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:20.880518 17466 tablet_server.cc:195] TabletServer@127.17.14.130:0 shutdown complete.
I20250109 20:59:20.890291 17466 tablet_server.cc:178] TabletServer@127.17.14.131:0 shutting down...
I20250109 20:59:20.910449 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:20.911064 17466 tablet_replica.cc:331] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5: stopping tablet replica
I20250109 20:59:20.911561 17466 raft_consensus.cc:2238] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:59:20.912027 17466 raft_consensus.cc:2267] T e84a6233b1d6435488e2baf8bbb34462 P 24d1d60a52a5406db7826d7a52c719e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:20.930325 17466 tablet_server.cc:195] TabletServer@127.17.14.131:0 shutdown complete.
I20250109 20:59:20.939869 17466 tablet_server.cc:178] TabletServer@127.17.14.132:0 shutting down...
I20250109 20:59:20.958397 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:20.989372 17466 tablet_server.cc:195] TabletServer@127.17.14.132:0 shutdown complete.
I20250109 20:59:20.997431 17466 master.cc:537] Master@127.17.14.190:42199 shutting down...
I20250109 20:59:21.015189 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:59:21.015807 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:21.016134 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P b5dc125bd5724e3c85f41de0aa07cd6e: stopping tablet replica
I20250109 20:59:21.034044 17466 master.cc:559] Master@127.17.14.190:42199 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (14258 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20250109 20:59:21.069321 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:42835
I20250109 20:59:21.070417 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:21.085768 20426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:21.086223 20425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:21.090094 17466 server_base.cc:1034] running on GCE node
W20250109 20:59:21.091035 20429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:21.091977 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:21.092172 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:21.092288 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456361092276 us; error 0 us; skew 500 ppm
I20250109 20:59:21.092852 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:21.095638 17466 webserver.cc:458] Webserver started at http://127.17.14.190:36649/ using document root <none> and password file <none>
I20250109 20:59:21.096179 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:21.096340 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:21.096606 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:21.098086 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "97ac8264cc8946588bfbcf7ea5d4f81a"
format_stamp: "Formatted at 2025-01-09 20:59:21 on dist-test-slave-j54z"
I20250109 20:59:21.102559 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250109 20:59:21.105953 20434 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:21.106673 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20250109 20:59:21.106925 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "97ac8264cc8946588bfbcf7ea5d4f81a"
format_stamp: "Formatted at 2025-01-09 20:59:21 on dist-test-slave-j54z"
I20250109 20:59:21.107209 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:21.116014 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:21.117218 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:21.155709 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:42835
I20250109 20:59:21.155802 20485 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:42835 every 8 connection(s)
I20250109 20:59:21.159775 20486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:21.171125 20486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a: Bootstrap starting.
I20250109 20:59:21.175770 20486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:21.180001 20486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a: No bootstrap required, opened a new log
I20250109 20:59:21.182138 20486 raft_consensus.cc:357] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ac8264cc8946588bfbcf7ea5d4f81a" member_type: VOTER }
I20250109 20:59:21.182647 20486 raft_consensus.cc:383] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:21.182878 20486 raft_consensus.cc:738] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ac8264cc8946588bfbcf7ea5d4f81a, State: Initialized, Role: FOLLOWER
I20250109 20:59:21.183485 20486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ac8264cc8946588bfbcf7ea5d4f81a" member_type: VOTER }
I20250109 20:59:21.183944 20486 raft_consensus.cc:397] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:59:21.184150 20486 raft_consensus.cc:491] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:59:21.184425 20486 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:21.188642 20486 raft_consensus.cc:513] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ac8264cc8946588bfbcf7ea5d4f81a" member_type: VOTER }
I20250109 20:59:21.189108 20486 leader_election.cc:304] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97ac8264cc8946588bfbcf7ea5d4f81a; no voters:
I20250109 20:59:21.190183 20486 leader_election.cc:290] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:59:21.190563 20489 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:59:21.191824 20489 raft_consensus.cc:695] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [term 1 LEADER]: Becoming Leader. State: Replica: 97ac8264cc8946588bfbcf7ea5d4f81a, State: Running, Role: LEADER
I20250109 20:59:21.192430 20489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ac8264cc8946588bfbcf7ea5d4f81a" member_type: VOTER }
I20250109 20:59:21.193226 20486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:59:21.199302 20491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97ac8264cc8946588bfbcf7ea5d4f81a. Latest consensus state: current_term: 1 leader_uuid: "97ac8264cc8946588bfbcf7ea5d4f81a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ac8264cc8946588bfbcf7ea5d4f81a" member_type: VOTER } }
I20250109 20:59:21.199772 20491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [sys.catalog]: This master's current role is: LEADER
I20250109 20:59:21.201452 20490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97ac8264cc8946588bfbcf7ea5d4f81a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ac8264cc8946588bfbcf7ea5d4f81a" member_type: VOTER } }
I20250109 20:59:21.201763 20495 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:59:21.202157 20490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [sys.catalog]: This master's current role is: LEADER
I20250109 20:59:21.210552 20495 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:59:21.211625 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 20:59:21.219372 20495 catalog_manager.cc:1348] Generated new cluster ID: ee6ef0f1ba4444b5b17f6420b7ccf8ed
I20250109 20:59:21.219677 20495 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:59:21.252032 20495 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 20:59:21.253253 20495 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:59:21.267828 20495 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a: Generated new TSK 0
I20250109 20:59:21.268416 20495 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:59:21.277861 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:21.284386 20507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:21.285948 20508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:21.286526 20510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:21.287493 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:21.288251 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:21.288443 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:21.288591 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456361288575 us; error 0 us; skew 500 ppm
I20250109 20:59:21.289067 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:21.291312 17466 webserver.cc:458] Webserver started at http://127.17.14.129:45191/ using document root <none> and password file <none>
I20250109 20:59:21.291764 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:21.291945 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:21.292193 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:21.293337 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-0-root/instance:
uuid: "5721b8d7465840fc8288c7f24212d70a"
format_stamp: "Formatted at 2025-01-09 20:59:21 on dist-test-slave-j54z"
I20250109 20:59:21.297873 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250109 20:59:21.300926 20515 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:21.301694 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 20:59:21.301918 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "5721b8d7465840fc8288c7f24212d70a"
format_stamp: "Formatted at 2025-01-09 20:59:21 on dist-test-slave-j54z"
I20250109 20:59:21.302140 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:21.313612 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:21.314698 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:21.316044 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:21.318346 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:21.318514 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:21.318683 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:21.318814 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:21.356652 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:43597
I20250109 20:59:21.356734 20577 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:43597 every 8 connection(s)
I20250109 20:59:21.361377 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:21.368942 20582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:21.370112 20583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:21.370885 20585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:21.372828 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:21.373894 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250109 20:59:21.373862 20578 heartbeater.cc:346] Connected to a master server at 127.17.14.190:42835
W20250109 20:59:21.374257 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:21.374411 20578 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:21.374434 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456361374418 us; error 0 us; skew 500 ppm
I20250109 20:59:21.375097 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:21.375173 20578 heartbeater.cc:510] Master 127.17.14.190:42835 requested a full tablet report, sending...
I20250109 20:59:21.377625 17466 webserver.cc:458] Webserver started at http://127.17.14.130:36553/ using document root <none> and password file <none>
I20250109 20:59:21.377584 20451 ts_manager.cc:194] Registered new tserver with Master: 5721b8d7465840fc8288c7f24212d70a (127.17.14.129:43597)
I20250109 20:59:21.378234 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:21.378549 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:21.378803 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:21.379613 20451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57406
I20250109 20:59:21.379890 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-1-root/instance:
uuid: "0b3fef18429b4534959007ff309558f2"
format_stamp: "Formatted at 2025-01-09 20:59:21 on dist-test-slave-j54z"
I20250109 20:59:21.384333 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 20:59:21.387331 20590 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:21.388031 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 20:59:21.388283 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "0b3fef18429b4534959007ff309558f2"
format_stamp: "Formatted at 2025-01-09 20:59:21 on dist-test-slave-j54z"
I20250109 20:59:21.388509 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:21.408347 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:21.409608 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:21.410993 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:21.413192 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:21.413362 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:21.413611 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:21.413750 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:21.450222 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:46407
I20250109 20:59:21.450320 20652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:46407 every 8 connection(s)
I20250109 20:59:21.454728 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:21.462102 20656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:21.463061 20657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:21.465893 20653 heartbeater.cc:346] Connected to a master server at 127.17.14.190:42835
W20250109 20:59:21.466028 20659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:21.466348 20653 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:21.467059 20653 heartbeater.cc:510] Master 127.17.14.190:42835 requested a full tablet report, sending...
I20250109 20:59:21.467482 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:21.468274 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:21.468516 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:21.468712 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456361468695 us; error 0 us; skew 500 ppm
I20250109 20:59:21.468999 20451 ts_manager.cc:194] Registered new tserver with Master: 0b3fef18429b4534959007ff309558f2 (127.17.14.130:46407)
I20250109 20:59:21.469372 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:21.470921 20451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57420
I20250109 20:59:21.472359 17466 webserver.cc:458] Webserver started at http://127.17.14.131:34297/ using document root <none> and password file <none>
I20250109 20:59:21.472997 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:21.473232 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:21.473512 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:21.474622 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-2-root/instance:
uuid: "738f2d5427ac40b395e4bf83e1384266"
format_stamp: "Formatted at 2025-01-09 20:59:21 on dist-test-slave-j54z"
I20250109 20:59:21.478793 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250109 20:59:21.481619 20664 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:21.482306 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:59:21.482527 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "738f2d5427ac40b395e4bf83e1384266"
format_stamp: "Formatted at 2025-01-09 20:59:21 on dist-test-slave-j54z"
I20250109 20:59:21.482753 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:21.508630 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:21.509881 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:21.511215 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:21.513340 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:21.513546 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:21.513769 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:21.513922 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:21.556344 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:33449
I20250109 20:59:21.556443 20726 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:33449 every 8 connection(s)
I20250109 20:59:21.574379 20727 heartbeater.cc:346] Connected to a master server at 127.17.14.190:42835
I20250109 20:59:21.574724 20727 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:21.575425 20727 heartbeater.cc:510] Master 127.17.14.190:42835 requested a full tablet report, sending...
I20250109 20:59:21.577071 20451 ts_manager.cc:194] Registered new tserver with Master: 738f2d5427ac40b395e4bf83e1384266 (127.17.14.131:33449)
I20250109 20:59:21.577976 17466 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018676362s
I20250109 20:59:21.578624 20451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57426
I20250109 20:59:22.381897 20578 heartbeater.cc:502] Master 127.17.14.190:42835 was elected leader, sending a full tablet report...
I20250109 20:59:22.473650 20653 heartbeater.cc:502] Master 127.17.14.190:42835 was elected leader, sending a full tablet report...
I20250109 20:59:22.581085 20727 heartbeater.cc:502] Master 127.17.14.190:42835 was elected leader, sending a full tablet report...
I20250109 20:59:22.620991 17466 test_util.cc:274] Using random seed: 1084848168
I20250109 20:59:22.641317 20451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57450:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:59:22.643141 20451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:59:22.692339 20692 tablet_service.cc:1467] Processing CreateTablet for tablet acb42c92fa3a4fecbb57b2bca9c801fa (DEFAULT_TABLE table=test-workload [id=f21dd76586374347943f80635c89e175]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:59:22.693729 20692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acb42c92fa3a4fecbb57b2bca9c801fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:22.699889 20618 tablet_service.cc:1467] Processing CreateTablet for tablet acb42c92fa3a4fecbb57b2bca9c801fa (DEFAULT_TABLE table=test-workload [id=f21dd76586374347943f80635c89e175]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:59:22.699895 20543 tablet_service.cc:1467] Processing CreateTablet for tablet acb42c92fa3a4fecbb57b2bca9c801fa (DEFAULT_TABLE table=test-workload [id=f21dd76586374347943f80635c89e175]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:59:22.701323 20618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acb42c92fa3a4fecbb57b2bca9c801fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:22.701309 20543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acb42c92fa3a4fecbb57b2bca9c801fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:22.710539 20748 tablet_bootstrap.cc:492] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266: Bootstrap starting.
I20250109 20:59:22.717310 20748 tablet_bootstrap.cc:654] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:22.718084 20749 tablet_bootstrap.cc:492] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a: Bootstrap starting.
I20250109 20:59:22.719883 20750 tablet_bootstrap.cc:492] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2: Bootstrap starting.
I20250109 20:59:22.725308 20749 tablet_bootstrap.cc:654] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:22.726874 20750 tablet_bootstrap.cc:654] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:22.727722 20748 tablet_bootstrap.cc:492] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266: No bootstrap required, opened a new log
I20250109 20:59:22.728132 20748 ts_tablet_manager.cc:1397] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20250109 20:59:22.730580 20748 raft_consensus.cc:357] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f2d5427ac40b395e4bf83e1384266" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33449 } } peers { permanent_uuid: "5721b8d7465840fc8288c7f24212d70a" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 43597 } } peers { permanent_uuid: "0b3fef18429b4534959007ff309558f2" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 46407 } }
I20250109 20:59:22.731146 20748 raft_consensus.cc:383] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:22.731379 20748 raft_consensus.cc:738] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738f2d5427ac40b395e4bf83e1384266, State: Initialized, Role: FOLLOWER
I20250109 20:59:22.731964 20748 consensus_queue.cc:260] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f2d5427ac40b395e4bf83e1384266" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33449 } } peers { permanent_uuid: "5721b8d7465840fc8288c7f24212d70a" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 43597 } } peers { permanent_uuid: "0b3fef18429b4534959007ff309558f2" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 46407 } }
I20250109 20:59:22.737804 20749 tablet_bootstrap.cc:492] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a: No bootstrap required, opened a new log
I20250109 20:59:22.738193 20748 ts_tablet_manager.cc:1428] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
I20250109 20:59:22.738365 20749 ts_tablet_manager.cc:1397] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.007s
I20250109 20:59:22.738583 20750 tablet_bootstrap.cc:492] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2: No bootstrap required, opened a new log
I20250109 20:59:22.739133 20750 ts_tablet_manager.cc:1397] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20250109 20:59:22.741307 20750 raft_consensus.cc:357] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f2d5427ac40b395e4bf83e1384266" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33449 } } peers { permanent_uuid: "5721b8d7465840fc8288c7f24212d70a" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 43597 } } peers { permanent_uuid: "0b3fef18429b4534959007ff309558f2" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 46407 } }
I20250109 20:59:22.741212 20749 raft_consensus.cc:357] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f2d5427ac40b395e4bf83e1384266" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33449 } } peers { permanent_uuid: "5721b8d7465840fc8288c7f24212d70a" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 43597 } } peers { permanent_uuid: "0b3fef18429b4534959007ff309558f2" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 46407 } }
I20250109 20:59:22.741957 20750 raft_consensus.cc:383] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:22.742102 20749 raft_consensus.cc:383] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:22.742267 20750 raft_consensus.cc:738] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3fef18429b4534959007ff309558f2, State: Initialized, Role: FOLLOWER
I20250109 20:59:22.742506 20749 raft_consensus.cc:738] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5721b8d7465840fc8288c7f24212d70a, State: Initialized, Role: FOLLOWER
I20250109 20:59:22.742960 20750 consensus_queue.cc:260] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f2d5427ac40b395e4bf83e1384266" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33449 } } peers { permanent_uuid: "5721b8d7465840fc8288c7f24212d70a" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 43597 } } peers { permanent_uuid: "0b3fef18429b4534959007ff309558f2" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 46407 } }
I20250109 20:59:22.743332 20749 consensus_queue.cc:260] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f2d5427ac40b395e4bf83e1384266" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33449 } } peers { permanent_uuid: "5721b8d7465840fc8288c7f24212d70a" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 43597 } } peers { permanent_uuid: "0b3fef18429b4534959007ff309558f2" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 46407 } }
I20250109 20:59:22.745887 20750 ts_tablet_manager.cc:1428] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250109 20:59:22.749446 20749 ts_tablet_manager.cc:1428] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20250109 20:59:22.941057 20756 raft_consensus.cc:491] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:59:22.941462 20756 raft_consensus.cc:513] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f2d5427ac40b395e4bf83e1384266" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33449 } } peers { permanent_uuid: "5721b8d7465840fc8288c7f24212d70a" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 43597 } } peers { permanent_uuid: "0b3fef18429b4534959007ff309558f2" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 46407 } }
I20250109 20:59:22.943288 20756 leader_election.cc:290] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 738f2d5427ac40b395e4bf83e1384266 (127.17.14.131:33449), 0b3fef18429b4534959007ff309558f2 (127.17.14.130:46407)
I20250109 20:59:22.953544 20702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb42c92fa3a4fecbb57b2bca9c801fa" candidate_uuid: "5721b8d7465840fc8288c7f24212d70a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738f2d5427ac40b395e4bf83e1384266" is_pre_election: true
I20250109 20:59:22.953541 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb42c92fa3a4fecbb57b2bca9c801fa" candidate_uuid: "5721b8d7465840fc8288c7f24212d70a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3fef18429b4534959007ff309558f2" is_pre_election: true
I20250109 20:59:22.954418 20702 raft_consensus.cc:2463] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5721b8d7465840fc8288c7f24212d70a in term 0.
I20250109 20:59:22.954525 20628 raft_consensus.cc:2463] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5721b8d7465840fc8288c7f24212d70a in term 0.
I20250109 20:59:22.955847 20518 leader_election.cc:304] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5721b8d7465840fc8288c7f24212d70a, 738f2d5427ac40b395e4bf83e1384266; no voters:
I20250109 20:59:22.956631 20756 raft_consensus.cc:2798] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:59:22.956952 20756 raft_consensus.cc:491] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:59:22.957171 20756 raft_consensus.cc:3054] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:22.961988 20756 raft_consensus.cc:513] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f2d5427ac40b395e4bf83e1384266" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33449 } } peers { permanent_uuid: "5721b8d7465840fc8288c7f24212d70a" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 43597 } } peers { permanent_uuid: "0b3fef18429b4534959007ff309558f2" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 46407 } }
I20250109 20:59:22.963485 20756 leader_election.cc:290] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [CANDIDATE]: Term 1 election: Requested vote from peers 738f2d5427ac40b395e4bf83e1384266 (127.17.14.131:33449), 0b3fef18429b4534959007ff309558f2 (127.17.14.130:46407)
I20250109 20:59:22.964177 20702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb42c92fa3a4fecbb57b2bca9c801fa" candidate_uuid: "5721b8d7465840fc8288c7f24212d70a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738f2d5427ac40b395e4bf83e1384266"
I20250109 20:59:22.964401 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb42c92fa3a4fecbb57b2bca9c801fa" candidate_uuid: "5721b8d7465840fc8288c7f24212d70a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3fef18429b4534959007ff309558f2"
I20250109 20:59:22.964639 20702 raft_consensus.cc:3054] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:22.964931 20628 raft_consensus.cc:3054] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:22.969354 20702 raft_consensus.cc:2463] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5721b8d7465840fc8288c7f24212d70a in term 1.
I20250109 20:59:22.969421 20628 raft_consensus.cc:2463] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5721b8d7465840fc8288c7f24212d70a in term 1.
I20250109 20:59:22.970345 20518 leader_election.cc:304] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5721b8d7465840fc8288c7f24212d70a, 738f2d5427ac40b395e4bf83e1384266; no voters:
I20250109 20:59:22.970947 20756 raft_consensus.cc:2798] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:59:22.972004 20756 raft_consensus.cc:695] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 1 LEADER]: Becoming Leader. State: Replica: 5721b8d7465840fc8288c7f24212d70a, State: Running, Role: LEADER
I20250109 20:59:22.972700 20756 consensus_queue.cc:237] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f2d5427ac40b395e4bf83e1384266" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33449 } } peers { permanent_uuid: "5721b8d7465840fc8288c7f24212d70a" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 43597 } } peers { permanent_uuid: "0b3fef18429b4534959007ff309558f2" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 46407 } }
I20250109 20:59:22.979789 20450 catalog_manager.cc:5526] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5721b8d7465840fc8288c7f24212d70a (127.17.14.129). New cstate: current_term: 1 leader_uuid: "5721b8d7465840fc8288c7f24212d70a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f2d5427ac40b395e4bf83e1384266" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5721b8d7465840fc8288c7f24212d70a" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 43597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b3fef18429b4534959007ff309558f2" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20250109 20:59:23.209174 20499 placement_policy_util.cc:407] tablet acb42c92fa3a4fecbb57b2bca9c801fa: detected majority of replicas (2 of 3) at location L0
I20250109 20:59:23.211318 20504 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:23.212579 20504 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:23.212868 20504 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:23.440603 20756 consensus_queue.cc:1035] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b3fef18429b4534959007ff309558f2" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:59:23.451476 20756 consensus_queue.cc:1035] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [LEADER]: Connected to new peer: Peer: permanent_uuid: "738f2d5427ac40b395e4bf83e1384266" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 33449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:59:24.217170 20499 placement_policy_util.cc:407] tablet acb42c92fa3a4fecbb57b2bca9c801fa: detected majority of replicas (2 of 3) at location L0
I20250109 20:59:25.213716 20504 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:25.214890 20504 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:25.215166 20504 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:25.224749 20499 placement_policy_util.cc:407] tablet acb42c92fa3a4fecbb57b2bca9c801fa: detected majority of replicas (2 of 3) at location L0
I20250109 20:59:26.232535 20499 placement_policy_util.cc:407] tablet acb42c92fa3a4fecbb57b2bca9c801fa: detected majority of replicas (2 of 3) at location L0
I20250109 20:59:27.216176 20504 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:27.217511 20504 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:27.217770 20504 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:27.240108 20499 placement_policy_util.cc:407] tablet acb42c92fa3a4fecbb57b2bca9c801fa: detected majority of replicas (2 of 3) at location L0
I20250109 20:59:28.247221 20499 placement_policy_util.cc:407] tablet acb42c92fa3a4fecbb57b2bca9c801fa: detected majority of replicas (2 of 3) at location L0
I20250109 20:59:29.218544 20504 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:29.219857 20504 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:29.220180 20504 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:29.254662 20499 placement_policy_util.cc:407] tablet acb42c92fa3a4fecbb57b2bca9c801fa: detected majority of replicas (2 of 3) at location L0
I20250109 20:59:30.262066 20499 placement_policy_util.cc:407] tablet acb42c92fa3a4fecbb57b2bca9c801fa: detected majority of replicas (2 of 3) at location L0
I20250109 20:59:31.221570 20504 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:31.223381 20504 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:31.223641 20504 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:31.269685 20499 placement_policy_util.cc:407] tablet acb42c92fa3a4fecbb57b2bca9c801fa: detected majority of replicas (2 of 3) at location L0
I20250109 20:59:32.276392 20499 placement_policy_util.cc:407] tablet acb42c92fa3a4fecbb57b2bca9c801fa: detected majority of replicas (2 of 3) at location L0
I20250109 20:59:33.224423 20504 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:33.225672 20504 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:33.225932 20504 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:33.282615 20499 placement_policy_util.cc:407] tablet acb42c92fa3a4fecbb57b2bca9c801fa: detected majority of replicas (2 of 3) at location L0
I20250109 20:59:34.059408 17466 tablet_server.cc:178] TabletServer@127.17.14.129:0 shutting down...
I20250109 20:59:34.084288 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:34.084991 17466 tablet_replica.cc:331] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a: stopping tablet replica
I20250109 20:59:34.085620 17466 raft_consensus.cc:2238] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:59:34.086580 17466 raft_consensus.cc:2267] T acb42c92fa3a4fecbb57b2bca9c801fa P 5721b8d7465840fc8288c7f24212d70a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:34.106350 17466 tablet_server.cc:195] TabletServer@127.17.14.129:0 shutdown complete.
I20250109 20:59:34.118033 17466 tablet_server.cc:178] TabletServer@127.17.14.130:0 shutting down...
I20250109 20:59:34.141638 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:34.142428 17466 tablet_replica.cc:331] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2: stopping tablet replica
I20250109 20:59:34.143162 17466 raft_consensus.cc:2238] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:59:34.143644 17466 raft_consensus.cc:2267] T acb42c92fa3a4fecbb57b2bca9c801fa P 0b3fef18429b4534959007ff309558f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:34.162266 17466 tablet_server.cc:195] TabletServer@127.17.14.130:0 shutdown complete.
I20250109 20:59:34.172497 17466 tablet_server.cc:178] TabletServer@127.17.14.131:0 shutting down...
I20250109 20:59:34.192425 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:34.193181 17466 tablet_replica.cc:331] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266: stopping tablet replica
I20250109 20:59:34.193791 17466 raft_consensus.cc:2238] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:59:34.194321 17466 raft_consensus.cc:2267] T acb42c92fa3a4fecbb57b2bca9c801fa P 738f2d5427ac40b395e4bf83e1384266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:34.212587 17466 tablet_server.cc:195] TabletServer@127.17.14.131:0 shutdown complete.
I20250109 20:59:34.222474 17466 master.cc:537] Master@127.17.14.190:42835 shutting down...
I20250109 20:59:34.242563 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:59:34.243146 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:34.243463 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P 97ac8264cc8946588bfbcf7ea5d4f81a: stopping tablet replica
I20250109 20:59:34.261803 17466 master.cc:559] Master@127.17.14.190:42835 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (13224 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:516: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] AutoRebalancerTest.TestMaxMovesPerServer (6 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20250109 20:59:34.299799 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:46305
I20250109 20:59:34.300915 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:34.306720 20774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:34.307883 20775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:34.309120 20777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:34.310187 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:34.310879 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:34.311070 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:34.311210 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456374311194 us; error 0 us; skew 500 ppm
I20250109 20:59:34.311688 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:34.314086 17466 webserver.cc:458] Webserver started at http://127.17.14.190:45347/ using document root <none> and password file <none>
I20250109 20:59:34.314524 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:34.314687 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:34.314978 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:34.316147 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "fd1179d89ac5492bbdf06b42cc3a5553"
format_stamp: "Formatted at 2025-01-09 20:59:34 on dist-test-slave-j54z"
I20250109 20:59:34.320398 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 20:59:34.323542 20782 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:34.324224 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250109 20:59:34.324491 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "fd1179d89ac5492bbdf06b42cc3a5553"
format_stamp: "Formatted at 2025-01-09 20:59:34 on dist-test-slave-j54z"
I20250109 20:59:34.324743 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:34.336125 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:34.337270 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:34.372819 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:46305
I20250109 20:59:34.372918 20833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:46305 every 8 connection(s)
I20250109 20:59:34.376581 20834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:34.387319 20834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553: Bootstrap starting.
I20250109 20:59:34.392056 20834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:34.396104 20834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553: No bootstrap required, opened a new log
I20250109 20:59:34.398185 20834 raft_consensus.cc:357] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1179d89ac5492bbdf06b42cc3a5553" member_type: VOTER }
I20250109 20:59:34.398666 20834 raft_consensus.cc:383] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:34.398859 20834 raft_consensus.cc:738] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1179d89ac5492bbdf06b42cc3a5553, State: Initialized, Role: FOLLOWER
I20250109 20:59:34.399408 20834 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1179d89ac5492bbdf06b42cc3a5553" member_type: VOTER }
I20250109 20:59:34.399838 20834 raft_consensus.cc:397] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:59:34.400039 20834 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:59:34.400267 20834 raft_consensus.cc:3054] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:34.404603 20834 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1179d89ac5492bbdf06b42cc3a5553" member_type: VOTER }
I20250109 20:59:34.405055 20834 leader_election.cc:304] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd1179d89ac5492bbdf06b42cc3a5553; no voters:
I20250109 20:59:34.406106 20834 leader_election.cc:290] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:59:34.406378 20837 raft_consensus.cc:2798] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:59:34.407537 20837 raft_consensus.cc:695] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [term 1 LEADER]: Becoming Leader. State: Replica: fd1179d89ac5492bbdf06b42cc3a5553, State: Running, Role: LEADER
I20250109 20:59:34.408126 20837 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1179d89ac5492bbdf06b42cc3a5553" member_type: VOTER }
I20250109 20:59:34.409070 20834 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:59:34.410998 20838 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd1179d89ac5492bbdf06b42cc3a5553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1179d89ac5492bbdf06b42cc3a5553" member_type: VOTER } }
I20250109 20:59:34.411011 20839 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd1179d89ac5492bbdf06b42cc3a5553. Latest consensus state: current_term: 1 leader_uuid: "fd1179d89ac5492bbdf06b42cc3a5553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1179d89ac5492bbdf06b42cc3a5553" member_type: VOTER } }
I20250109 20:59:34.411697 20839 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [sys.catalog]: This master's current role is: LEADER
I20250109 20:59:34.413138 20838 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [sys.catalog]: This master's current role is: LEADER
I20250109 20:59:34.417711 20842 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:59:34.423450 20842 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:59:34.424342 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 20:59:34.431536 20842 catalog_manager.cc:1348] Generated new cluster ID: d269300de19645949f35025e38825527
I20250109 20:59:34.431840 20842 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:59:34.455598 20842 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 20:59:34.456863 20842 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:59:34.467522 20842 catalog_manager.cc:5899] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553: Generated new TSK 0
I20250109 20:59:34.468073 20842 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:59:34.490885 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:34.497110 20855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:34.498314 20856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:34.499097 20858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:34.500092 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:34.500823 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:34.501024 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:34.501171 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456374501155 us; error 0 us; skew 500 ppm
I20250109 20:59:34.501675 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:34.503926 17466 webserver.cc:458] Webserver started at http://127.17.14.129:39695/ using document root <none> and password file <none>
I20250109 20:59:34.504411 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:34.504583 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:34.504819 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:34.505952 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-0-root/instance:
uuid: "4dde147b5e4e4c20b5b3c20fedf7d576"
format_stamp: "Formatted at 2025-01-09 20:59:34 on dist-test-slave-j54z"
I20250109 20:59:34.510265 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20250109 20:59:34.513284 20863 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:34.513967 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 20:59:34.514221 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "4dde147b5e4e4c20b5b3c20fedf7d576"
format_stamp: "Formatted at 2025-01-09 20:59:34 on dist-test-slave-j54z"
I20250109 20:59:34.514487 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:34.543428 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:34.544538 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:34.545925 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:34.548097 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:34.548332 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:34.548537 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:34.548679 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:34.586154 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:38533
I20250109 20:59:34.586249 20925 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:38533 every 8 connection(s)
I20250109 20:59:34.590586 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:34.597890 20930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:34.599298 20931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:34.606062 20933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:34.606300 20926 heartbeater.cc:346] Connected to a master server at 127.17.14.190:46305
I20250109 20:59:34.606663 20926 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:34.607378 20926 heartbeater.cc:510] Master 127.17.14.190:46305 requested a full tablet report, sending...
I20250109 20:59:34.607477 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:34.608275 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:34.608520 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:34.608723 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456374608705 us; error 0 us; skew 500 ppm
I20250109 20:59:34.609215 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:34.609386 20799 ts_manager.cc:194] Registered new tserver with Master: 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533)
I20250109 20:59:34.611469 20799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42836
I20250109 20:59:34.611857 17466 webserver.cc:458] Webserver started at http://127.17.14.130:38747/ using document root <none> and password file <none>
I20250109 20:59:34.612351 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:34.612587 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:34.612841 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:34.614007 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-1-root/instance:
uuid: "25d0989f1b044c549bec7fb6f03ca272"
format_stamp: "Formatted at 2025-01-09 20:59:34 on dist-test-slave-j54z"
I20250109 20:59:34.618142 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250109 20:59:34.621158 20938 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:34.621858 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 20:59:34.622114 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "25d0989f1b044c549bec7fb6f03ca272"
format_stamp: "Formatted at 2025-01-09 20:59:34 on dist-test-slave-j54z"
I20250109 20:59:34.622388 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:34.640141 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:34.641237 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:34.642596 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:34.644647 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:34.644836 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:34.645038 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:34.645184 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:34.682535 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:34503
I20250109 20:59:34.682646 21000 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:34503 every 8 connection(s)
I20250109 20:59:34.686864 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:34.693967 21004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:34.695205 21005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:34.696349 21007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:34.698029 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:34.698225 21001 heartbeater.cc:346] Connected to a master server at 127.17.14.190:46305
I20250109 20:59:34.698607 21001 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:34.698885 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:34.699090 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:34.699259 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456374699241 us; error 0 us; skew 500 ppm
I20250109 20:59:34.699303 21001 heartbeater.cc:510] Master 127.17.14.190:46305 requested a full tablet report, sending...
I20250109 20:59:34.699772 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:34.701191 20799 ts_manager.cc:194] Registered new tserver with Master: 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503)
I20250109 20:59:34.702307 17466 webserver.cc:458] Webserver started at http://127.17.14.131:35629/ using document root <none> and password file <none>
I20250109 20:59:34.702670 20799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42848
I20250109 20:59:34.702787 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:34.703048 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:34.703285 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:34.704416 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-2-root/instance:
uuid: "3ba4ac9555ec4772814224be02daee66"
format_stamp: "Formatted at 2025-01-09 20:59:34 on dist-test-slave-j54z"
I20250109 20:59:34.708592 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 20:59:34.711650 21012 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:34.712344 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 20:59:34.712594 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "3ba4ac9555ec4772814224be02daee66"
format_stamp: "Formatted at 2025-01-09 20:59:34 on dist-test-slave-j54z"
I20250109 20:59:34.712842 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:34.727865 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:34.728927 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:34.730260 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:34.732311 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:34.732498 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:34.732707 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:34.732857 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:34.769953 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:35083
I20250109 20:59:34.770020 21074 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:35083 every 8 connection(s)
I20250109 20:59:34.782541 21075 heartbeater.cc:346] Connected to a master server at 127.17.14.190:46305
I20250109 20:59:34.782946 21075 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:34.783581 21075 heartbeater.cc:510] Master 127.17.14.190:46305 requested a full tablet report, sending...
I20250109 20:59:34.785255 20799 ts_manager.cc:194] Registered new tserver with Master: 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:34.786227 17466 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013223989s
I20250109 20:59:34.786526 20799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42850
I20250109 20:59:35.613914 20926 heartbeater.cc:502] Master 127.17.14.190:46305 was elected leader, sending a full tablet report...
I20250109 20:59:35.704818 21001 heartbeater.cc:502] Master 127.17.14.190:46305 was elected leader, sending a full tablet report...
I20250109 20:59:35.789152 21075 heartbeater.cc:502] Master 127.17.14.190:46305 was elected leader, sending a full tablet report...
I20250109 20:59:35.817953 17466 test_util.cc:274] Using random seed: 1098045110
I20250109 20:59:35.843832 20799 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42862:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:59:35.846115 20799 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:59:35.890093 21040 tablet_service.cc:1467] Processing CreateTablet for tablet 65c1b0368bbb4c9eb5768d4412894583 (DEFAULT_TABLE table=test-workload [id=72ff5ee9e9764c7ead5d3d2eaeb14842]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250109 20:59:35.891554 21040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c1b0368bbb4c9eb5768d4412894583. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:35.891214 21039 tablet_service.cc:1467] Processing CreateTablet for tablet 060814a37ffc4435a40559089a4c4172 (DEFAULT_TABLE table=test-workload [id=72ff5ee9e9764c7ead5d3d2eaeb14842]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250109 20:59:35.892518 21039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 060814a37ffc4435a40559089a4c4172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:35.897432 20966 tablet_service.cc:1467] Processing CreateTablet for tablet 65c1b0368bbb4c9eb5768d4412894583 (DEFAULT_TABLE table=test-workload [id=72ff5ee9e9764c7ead5d3d2eaeb14842]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250109 20:59:35.898715 20966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c1b0368bbb4c9eb5768d4412894583. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:35.902556 20965 tablet_service.cc:1467] Processing CreateTablet for tablet 060814a37ffc4435a40559089a4c4172 (DEFAULT_TABLE table=test-workload [id=72ff5ee9e9764c7ead5d3d2eaeb14842]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250109 20:59:35.904119 21038 tablet_service.cc:1467] Processing CreateTablet for tablet f5dcf8c0f9e74e39ba0bd00de847ef6d (DEFAULT_TABLE table=test-workload [id=72ff5ee9e9764c7ead5d3d2eaeb14842]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250109 20:59:35.905409 21038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5dcf8c0f9e74e39ba0bd00de847ef6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:35.905823 20964 tablet_service.cc:1467] Processing CreateTablet for tablet f5dcf8c0f9e74e39ba0bd00de847ef6d (DEFAULT_TABLE table=test-workload [id=72ff5ee9e9764c7ead5d3d2eaeb14842]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250109 20:59:35.904134 20965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 060814a37ffc4435a40559089a4c4172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:35.908331 20964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5dcf8c0f9e74e39ba0bd00de847ef6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:35.925980 20891 tablet_service.cc:1467] Processing CreateTablet for tablet 65c1b0368bbb4c9eb5768d4412894583 (DEFAULT_TABLE table=test-workload [id=72ff5ee9e9764c7ead5d3d2eaeb14842]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250109 20:59:35.926666 20890 tablet_service.cc:1467] Processing CreateTablet for tablet 060814a37ffc4435a40559089a4c4172 (DEFAULT_TABLE table=test-workload [id=72ff5ee9e9764c7ead5d3d2eaeb14842]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250109 20:59:35.927397 20891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c1b0368bbb4c9eb5768d4412894583. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:35.928848 20889 tablet_service.cc:1467] Processing CreateTablet for tablet f5dcf8c0f9e74e39ba0bd00de847ef6d (DEFAULT_TABLE table=test-workload [id=72ff5ee9e9764c7ead5d3d2eaeb14842]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250109 20:59:35.930109 20889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5dcf8c0f9e74e39ba0bd00de847ef6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:35.936775 20890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 060814a37ffc4435a40559089a4c4172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:35.954885 21095 tablet_bootstrap.cc:492] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66: Bootstrap starting.
I20250109 20:59:35.961000 21095 tablet_bootstrap.cc:654] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:35.961339 21096 tablet_bootstrap.cc:492] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272: Bootstrap starting.
I20250109 20:59:35.969182 21096 tablet_bootstrap.cc:654] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:35.970912 21097 tablet_bootstrap.cc:492] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576: Bootstrap starting.
I20250109 20:59:35.973568 21095 tablet_bootstrap.cc:492] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66: No bootstrap required, opened a new log
I20250109 20:59:35.974143 21095 ts_tablet_manager.cc:1397] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20250109 20:59:35.977104 21095 raft_consensus.cc:357] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:35.977743 21097 tablet_bootstrap.cc:654] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:35.978055 21095 raft_consensus.cc:383] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:35.978379 21095 raft_consensus.cc:738] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba4ac9555ec4772814224be02daee66, State: Initialized, Role: FOLLOWER
I20250109 20:59:35.979046 21096 tablet_bootstrap.cc:492] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272: No bootstrap required, opened a new log
I20250109 20:59:35.979429 21096 ts_tablet_manager.cc:1397] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20250109 20:59:35.979125 21095 consensus_queue.cc:260] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:35.981652 21096 raft_consensus.cc:357] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:35.982389 21095 ts_tablet_manager.cc:1428] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20250109 20:59:35.982625 21096 raft_consensus.cc:383] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:35.983000 21096 raft_consensus.cc:738] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d0989f1b044c549bec7fb6f03ca272, State: Initialized, Role: FOLLOWER
I20250109 20:59:35.983625 21095 tablet_bootstrap.cc:492] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66: Bootstrap starting.
I20250109 20:59:35.983712 21096 consensus_queue.cc:260] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:35.985414 21097 tablet_bootstrap.cc:492] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576: No bootstrap required, opened a new log
I20250109 20:59:35.985932 21097 ts_tablet_manager.cc:1397] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20250109 20:59:35.988775 21097 raft_consensus.cc:357] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:35.989588 21097 raft_consensus.cc:383] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:35.989905 21097 raft_consensus.cc:738] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dde147b5e4e4c20b5b3c20fedf7d576, State: Initialized, Role: FOLLOWER
I20250109 20:59:35.990386 21095 tablet_bootstrap.cc:654] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:35.990620 21097 consensus_queue.cc:260] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:35.995635 21096 ts_tablet_manager.cc:1428] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272: Time spent starting tablet: real 0.016s user 0.010s sys 0.003s
I20250109 20:59:35.996475 21096 tablet_bootstrap.cc:492] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272: Bootstrap starting.
I20250109 20:59:36.002449 21096 tablet_bootstrap.cc:654] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:36.002518 21095 tablet_bootstrap.cc:492] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66: No bootstrap required, opened a new log
I20250109 20:59:36.003799 21095 ts_tablet_manager.cc:1397] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.003s
I20250109 20:59:36.008884 21095 raft_consensus.cc:357] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.009493 21097 ts_tablet_manager.cc:1428] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576: Time spent starting tablet: real 0.023s user 0.017s sys 0.004s
I20250109 20:59:36.009768 21095 raft_consensus.cc:383] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:36.010228 21095 raft_consensus.cc:738] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba4ac9555ec4772814224be02daee66, State: Initialized, Role: FOLLOWER
I20250109 20:59:36.010455 21097 tablet_bootstrap.cc:492] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576: Bootstrap starting.
I20250109 20:59:36.011047 21095 consensus_queue.cc:260] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.013345 21095 ts_tablet_manager.cc:1428] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250109 20:59:36.014393 21095 tablet_bootstrap.cc:492] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66: Bootstrap starting.
I20250109 20:59:36.017397 21097 tablet_bootstrap.cc:654] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:36.020613 21095 tablet_bootstrap.cc:654] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:36.024367 21096 tablet_bootstrap.cc:492] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272: No bootstrap required, opened a new log
I20250109 20:59:36.028132 21096 ts_tablet_manager.cc:1397] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.009s
I20250109 20:59:36.029330 21097 tablet_bootstrap.cc:492] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576: No bootstrap required, opened a new log
I20250109 20:59:36.029886 21097 ts_tablet_manager.cc:1397] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.007s
I20250109 20:59:36.030836 21096 raft_consensus.cc:357] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.031630 21096 raft_consensus.cc:383] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:36.031931 21096 raft_consensus.cc:738] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d0989f1b044c549bec7fb6f03ca272, State: Initialized, Role: FOLLOWER
I20250109 20:59:36.032909 21095 tablet_bootstrap.cc:492] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66: No bootstrap required, opened a new log
I20250109 20:59:36.032519 21097 raft_consensus.cc:357] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:36.032646 21096 consensus_queue.cc:260] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.033298 21095 ts_tablet_manager.cc:1397] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.000s
I20250109 20:59:36.033308 21097 raft_consensus.cc:383] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:36.033905 21097 raft_consensus.cc:738] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dde147b5e4e4c20b5b3c20fedf7d576, State: Initialized, Role: FOLLOWER
I20250109 20:59:36.035034 21096 ts_tablet_manager.cc:1428] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250109 20:59:36.034840 21097 consensus_queue.cc:260] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:36.035979 21096 tablet_bootstrap.cc:492] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272: Bootstrap starting.
I20250109 20:59:36.036182 21095 raft_consensus.cc:357] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.036964 21095 raft_consensus.cc:383] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:36.037281 21095 raft_consensus.cc:738] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba4ac9555ec4772814224be02daee66, State: Initialized, Role: FOLLOWER
I20250109 20:59:36.038121 21095 consensus_queue.cc:260] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.039378 21097 ts_tablet_manager.cc:1428] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250109 20:59:36.040408 21097 tablet_bootstrap.cc:492] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576: Bootstrap starting.
I20250109 20:59:36.042615 21096 tablet_bootstrap.cc:654] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:36.043219 21095 ts_tablet_manager.cc:1428] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250109 20:59:36.049858 21097 tablet_bootstrap.cc:654] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:36.055524 21097 tablet_bootstrap.cc:492] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576: No bootstrap required, opened a new log
I20250109 20:59:36.055971 21097 ts_tablet_manager.cc:1397] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20250109 20:59:36.056074 21096 tablet_bootstrap.cc:492] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272: No bootstrap required, opened a new log
I20250109 20:59:36.056541 21096 ts_tablet_manager.cc:1397] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.012s
I20250109 20:59:36.058669 21097 raft_consensus.cc:357] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.058933 21096 raft_consensus.cc:357] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:36.059473 21097 raft_consensus.cc:383] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:36.059607 21096 raft_consensus.cc:383] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:36.059805 21097 raft_consensus.cc:738] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dde147b5e4e4c20b5b3c20fedf7d576, State: Initialized, Role: FOLLOWER
I20250109 20:59:36.059924 21096 raft_consensus.cc:738] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d0989f1b044c549bec7fb6f03ca272, State: Initialized, Role: FOLLOWER
I20250109 20:59:36.060616 21096 consensus_queue.cc:260] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:36.060582 21097 consensus_queue.cc:260] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.063012 21096 ts_tablet_manager.cc:1428] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20250109 20:59:36.063010 21097 ts_tablet_manager.cc:1428] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250109 20:59:36.102025 21102 raft_consensus.cc:491] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:59:36.102509 21102 raft_consensus.cc:513] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.104485 21102 leader_election.cc:290] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533), 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:36.114682 20901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" is_pre_election: true
I20250109 20:59:36.114677 21050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66" is_pre_election: true
I20250109 20:59:36.115505 20901 raft_consensus.cc:2463] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 0.
I20250109 20:59:36.115586 21050 raft_consensus.cc:2463] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 0.
I20250109 20:59:36.116667 20941 leader_election.cc:304] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25d0989f1b044c549bec7fb6f03ca272, 3ba4ac9555ec4772814224be02daee66, 4dde147b5e4e4c20b5b3c20fedf7d576; no voters:
I20250109 20:59:36.117388 21102 raft_consensus.cc:2798] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:59:36.117741 21102 raft_consensus.cc:491] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:59:36.117998 21102 raft_consensus.cc:3054] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:36.122695 21102 raft_consensus.cc:513] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.124121 21102 leader_election.cc:290] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 1 election: Requested vote from peers 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533), 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:36.124939 20901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576"
I20250109 20:59:36.125068 21050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66"
I20250109 20:59:36.125402 20901 raft_consensus.cc:3054] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:36.125571 21050 raft_consensus.cc:3054] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:36.129913 20901 raft_consensus.cc:2463] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 1.
I20250109 20:59:36.130038 21050 raft_consensus.cc:2463] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 1.
I20250109 20:59:36.130769 20941 leader_election.cc:304] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25d0989f1b044c549bec7fb6f03ca272, 4dde147b5e4e4c20b5b3c20fedf7d576; no voters:
I20250109 20:59:36.131386 21102 raft_consensus.cc:2798] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:59:36.132431 21102 raft_consensus.cc:695] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Becoming Leader. State: Replica: 25d0989f1b044c549bec7fb6f03ca272, State: Running, Role: LEADER
I20250109 20:59:36.133112 21102 consensus_queue.cc:237] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.140403 20798 catalog_manager.cc:5526] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130). New cstate: current_term: 1 leader_uuid: "25d0989f1b044c549bec7fb6f03ca272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } health_report { overall_health: UNKNOWN } } }
I20250109 20:59:36.158318 21100 raft_consensus.cc:491] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:59:36.158739 21100 raft_consensus.cc:513] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:36.160727 21100 leader_election.cc:290] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503), 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533)
I20250109 20:59:36.163671 21102 raft_consensus.cc:491] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:59:36.164215 21102 raft_consensus.cc:513] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:36.166955 21050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66" is_pre_election: true
I20250109 20:59:36.167076 21102 leader_election.cc:290] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083), 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533)
I20250109 20:59:36.167294 20901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" is_pre_election: true
I20250109 20:59:36.167608 21050 raft_consensus.cc:2463] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 0.
I20250109 20:59:36.167987 20901 raft_consensus.cc:2463] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 0.
I20250109 20:59:36.168745 20939 leader_election.cc:304] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25d0989f1b044c549bec7fb6f03ca272, 3ba4ac9555ec4772814224be02daee66; no voters:
I20250109 20:59:36.169849 21102 raft_consensus.cc:2798] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:59:36.170192 21102 raft_consensus.cc:491] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:59:36.170572 21102 raft_consensus.cc:3054] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:36.173207 20976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "3ba4ac9555ec4772814224be02daee66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25d0989f1b044c549bec7fb6f03ca272" is_pre_election: true
I20250109 20:59:36.173425 20901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "3ba4ac9555ec4772814224be02daee66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" is_pre_election: true
I20250109 20:59:36.173995 20901 raft_consensus.cc:2463] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba4ac9555ec4772814224be02daee66 in term 0.
I20250109 20:59:36.174805 21015 leader_election.cc:304] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba4ac9555ec4772814224be02daee66, 4dde147b5e4e4c20b5b3c20fedf7d576; no voters:
I20250109 20:59:36.175420 21100 raft_consensus.cc:2798] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:59:36.175771 21100 raft_consensus.cc:491] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:59:36.176080 21100 raft_consensus.cc:3054] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:36.177018 21102 raft_consensus.cc:513] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:36.178167 20976 raft_consensus.cc:2388] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3ba4ac9555ec4772814224be02daee66 in current term 1: Already voted for candidate 25d0989f1b044c549bec7fb6f03ca272 in this term.
I20250109 20:59:36.179095 21103 raft_consensus.cc:491] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:59:36.179628 21103 raft_consensus.cc:513] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.179708 21102 leader_election.cc:290] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 1 election: Requested vote from peers 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083), 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533)
I20250109 20:59:36.180678 21102 raft_consensus.cc:491] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:59:36.180708 20901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576"
I20250109 20:59:36.181406 20901 raft_consensus.cc:3054] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:36.181840 21050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66"
I20250109 20:59:36.182190 21103 leader_election.cc:290] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503), 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:36.186450 21100 raft_consensus.cc:513] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:36.187645 21050 raft_consensus.cc:2388] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 25d0989f1b044c549bec7fb6f03ca272 in current term 1: Already voted for candidate 3ba4ac9555ec4772814224be02daee66 in this term.
I20250109 20:59:36.181195 21102 raft_consensus.cc:513] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.190702 20901 raft_consensus.cc:2463] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 1.
I20250109 20:59:36.191807 20900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060814a37ffc4435a40559089a4c4172" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" is_pre_election: true
I20250109 20:59:36.192705 20900 raft_consensus.cc:2463] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 0.
I20250109 20:59:36.192912 20976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "3ba4ac9555ec4772814224be02daee66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25d0989f1b044c549bec7fb6f03ca272"
I20250109 20:59:36.193550 20976 raft_consensus.cc:2388] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba4ac9555ec4772814224be02daee66 in current term 1: Already voted for candidate 25d0989f1b044c549bec7fb6f03ca272 in this term.
I20250109 20:59:36.193897 20941 leader_election.cc:304] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25d0989f1b044c549bec7fb6f03ca272, 4dde147b5e4e4c20b5b3c20fedf7d576; no voters: 3ba4ac9555ec4772814224be02daee66
I20250109 20:59:36.194839 21112 raft_consensus.cc:2798] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:59:36.194989 21102 leader_election.cc:290] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533), 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:36.195151 20941 leader_election.cc:304] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25d0989f1b044c549bec7fb6f03ca272, 4dde147b5e4e4c20b5b3c20fedf7d576; no voters:
I20250109 20:59:36.196645 21102 raft_consensus.cc:2798] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:59:36.197103 21102 raft_consensus.cc:491] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:59:36.199501 21102 raft_consensus.cc:3054] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:36.197000 21050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060814a37ffc4435a40559089a4c4172" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66" is_pre_election: true
I20250109 20:59:36.200771 21050 raft_consensus.cc:2463] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 0.
I20250109 20:59:36.202405 20900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "3ba4ac9555ec4772814224be02daee66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576"
I20250109 20:59:36.202767 21100 leader_election.cc:290] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [CANDIDATE]: Term 1 election: Requested vote from peers 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503), 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533)
I20250109 20:59:36.203159 20900 raft_consensus.cc:2388] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba4ac9555ec4772814224be02daee66 in current term 1: Already voted for candidate 25d0989f1b044c549bec7fb6f03ca272 in this term.
I20250109 20:59:36.204068 21015 leader_election.cc:304] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba4ac9555ec4772814224be02daee66; no voters: 25d0989f1b044c549bec7fb6f03ca272, 4dde147b5e4e4c20b5b3c20fedf7d576
I20250109 20:59:36.204655 21100 raft_consensus.cc:2743] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250109 20:59:36.207547 20976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060814a37ffc4435a40559089a4c4172" candidate_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25d0989f1b044c549bec7fb6f03ca272" is_pre_election: true
I20250109 20:59:36.210794 21112 raft_consensus.cc:695] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Becoming Leader. State: Replica: 25d0989f1b044c549bec7fb6f03ca272, State: Running, Role: LEADER
I20250109 20:59:36.211675 21112 consensus_queue.cc:237] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:36.214800 21102 raft_consensus.cc:513] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.216068 20976 raft_consensus.cc:2388] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4dde147b5e4e4c20b5b3c20fedf7d576 in current term 1: Already voted for candidate 25d0989f1b044c549bec7fb6f03ca272 in this term.
I20250109 20:59:36.218151 21050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060814a37ffc4435a40559089a4c4172" candidate_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66" is_pre_election: true
I20250109 20:59:36.218744 21050 raft_consensus.cc:2463] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dde147b5e4e4c20b5b3c20fedf7d576 in term 0.
I20250109 20:59:36.218830 20900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060814a37ffc4435a40559089a4c4172" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576"
I20250109 20:59:36.219457 20900 raft_consensus.cc:3054] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:36.219921 21050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060814a37ffc4435a40559089a4c4172" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66"
I20250109 20:59:36.220315 20864 leader_election.cc:304] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba4ac9555ec4772814224be02daee66, 4dde147b5e4e4c20b5b3c20fedf7d576; no voters:
I20250109 20:59:36.220600 21050 raft_consensus.cc:3054] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:36.221104 21102 leader_election.cc:290] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 1 election: Requested vote from peers 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533), 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:36.219581 20798 catalog_manager.cc:5526] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130). New cstate: current_term: 1 leader_uuid: "25d0989f1b044c549bec7fb6f03ca272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } health_report { overall_health: UNKNOWN } } }
I20250109 20:59:36.225309 20900 raft_consensus.cc:2463] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 1.
I20250109 20:59:36.225818 21103 raft_consensus.cc:2758] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250109 20:59:36.226255 21050 raft_consensus.cc:2463] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 1.
I20250109 20:59:36.227289 20939 leader_election.cc:304] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25d0989f1b044c549bec7fb6f03ca272, 3ba4ac9555ec4772814224be02daee66; no voters:
I20250109 20:59:36.227995 21102 raft_consensus.cc:2798] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:59:36.228363 21102 raft_consensus.cc:695] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Becoming Leader. State: Replica: 25d0989f1b044c549bec7fb6f03ca272, State: Running, Role: LEADER
I20250109 20:59:36.228977 21102 consensus_queue.cc:237] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.235210 20798 catalog_manager.cc:5526] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130). New cstate: current_term: 1 leader_uuid: "25d0989f1b044c549bec7fb6f03ca272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } health_report { overall_health: UNKNOWN } } }
I20250109 20:59:36.266070 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:36.272610 21121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:36.273959 21122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:36.275288 17466 server_base.cc:1034] running on GCE node
W20250109 20:59:36.275822 21124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:36.276906 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:36.277096 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:36.277276 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456376277254 us; error 0 us; skew 500 ppm
I20250109 20:59:36.277753 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:36.280320 17466 webserver.cc:458] Webserver started at http://127.17.14.132:46669/ using document root <none> and password file <none>
I20250109 20:59:36.280746 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:36.280927 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:36.281208 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:36.282346 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-3-root/instance:
uuid: "3eede40835234fef911414d9c515e30a"
format_stamp: "Formatted at 2025-01-09 20:59:36 on dist-test-slave-j54z"
I20250109 20:59:36.286814 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250109 20:59:36.289917 21129 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:36.290604 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 20:59:36.290881 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-3-root
uuid: "3eede40835234fef911414d9c515e30a"
format_stamp: "Formatted at 2025-01-09 20:59:36 on dist-test-slave-j54z"
I20250109 20:59:36.291153 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:36.299916 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:36.301038 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:36.302400 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:36.304562 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:36.304764 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:36.304960 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:36.305102 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:36.344316 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.132:43475
I20250109 20:59:36.344406 21191 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.132:43475 every 8 connection(s)
I20250109 20:59:36.357632 21192 heartbeater.cc:346] Connected to a master server at 127.17.14.190:46305
I20250109 20:59:36.357947 21192 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:36.358603 21192 heartbeater.cc:510] Master 127.17.14.190:46305 requested a full tablet report, sending...
I20250109 20:59:36.360252 20798 ts_manager.cc:194] Registered new tserver with Master: 3eede40835234fef911414d9c515e30a (127.17.14.132:43475)
I20250109 20:59:36.361593 20798 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42870
I20250109 20:59:36.423046 20852 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:36.437283 20976 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583"
dest_uuid: "25d0989f1b044c549bec7fb6f03ca272"
mode: GRACEFUL
new_leader_uuid: "3ba4ac9555ec4772814224be02daee66"
from {username='slave'} at 127.0.0.1:52682
I20250109 20:59:36.438006 20976 raft_consensus.cc:604] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Received request to transfer leadership to TS 3ba4ac9555ec4772814224be02daee66
W20250109 20:59:36.440115 20847 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20250109 20:59:36.440521 20976 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d"
dest_uuid: "25d0989f1b044c549bec7fb6f03ca272"
mode: GRACEFUL
new_leader_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576"
from {username='slave'} at 127.0.0.1:52682
I20250109 20:59:36.440989 20976 raft_consensus.cc:604] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Received request to transfer leadership to TS 4dde147b5e4e4c20b5b3c20fedf7d576
I20250109 20:59:36.442123 20852 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250109 20:59:36.442478 20852 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250109 20:59:36.443435 20847 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 060814a37ffc4435a40559089a4c4172, TS 4dde147b5e4e4c20b5b3c20fedf7d576 -> TS 3eede40835234fef911414d9c515e30a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
W20250109 20:59:36.443706 20847 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 060814a37ffc4435a40559089a4c4172, TS 4dde147b5e4e4c20b5b3c20fedf7d576 -> TS 3eede40835234fef911414d9c515e30a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
W20250109 20:59:36.446653 20847 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f5dcf8c0f9e74e39ba0bd00de847ef6d, TS 25d0989f1b044c549bec7fb6f03ca272 -> TS 3eede40835234fef911414d9c515e30a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
W20250109 20:59:36.446894 20847 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f5dcf8c0f9e74e39ba0bd00de847ef6d, TS 25d0989f1b044c549bec7fb6f03ca272 -> TS 3eede40835234fef911414d9c515e30a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.463032 21102 consensus_queue.cc:1035] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 20:59:36.469089 21112 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250109 20:59:36.469331 21112 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250109 20:59:36.485299 21118 consensus_queue.cc:1035] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250109 20:59:36.487393 21118 consensus_queue.cc:1035] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:59:36.523691 21102 raft_consensus.cc:988] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272: : Instructing follower 4dde147b5e4e4c20b5b3c20fedf7d576 to start an election
I20250109 20:59:36.524215 21118 raft_consensus.cc:1076] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Signalling peer 4dde147b5e4e4c20b5b3c20fedf7d576 to start an election
I20250109 20:59:36.525575 20900 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d"
dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576"
from {username='slave'} at 127.0.0.1:59698
I20250109 20:59:36.526039 20900 raft_consensus.cc:491] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250109 20:59:36.526319 20900 raft_consensus.cc:3054] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:59:36.530917 20900 raft_consensus.cc:513] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.532555 20900 leader_election.cc:290] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [CANDIDATE]: Term 2 election: Requested vote from peers 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503), 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:36.533272 20976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "25d0989f1b044c549bec7fb6f03ca272"
I20250109 20:59:36.533468 21050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3ba4ac9555ec4772814224be02daee66"
I20250109 20:59:36.533835 20976 raft_consensus.cc:3049] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Stepping down as leader of term 1
I20250109 20:59:36.533995 21050 raft_consensus.cc:3054] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:59:36.534137 20976 raft_consensus.cc:738] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 25d0989f1b044c549bec7fb6f03ca272, State: Running, Role: LEADER
I20250109 20:59:36.534808 20976 consensus_queue.cc:260] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.535661 20976 raft_consensus.cc:3054] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:59:36.538300 21050 raft_consensus.cc:2463] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dde147b5e4e4c20b5b3c20fedf7d576 in term 2.
I20250109 20:59:36.539073 20864 leader_election.cc:304] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba4ac9555ec4772814224be02daee66, 4dde147b5e4e4c20b5b3c20fedf7d576; no voters:
I20250109 20:59:36.539721 21103 raft_consensus.cc:2798] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 20:59:36.540329 20976 raft_consensus.cc:2463] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dde147b5e4e4c20b5b3c20fedf7d576 in term 2.
I20250109 20:59:36.540661 21103 raft_consensus.cc:695] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 LEADER]: Becoming Leader. State: Replica: 4dde147b5e4e4c20b5b3c20fedf7d576, State: Running, Role: LEADER
I20250109 20:59:36.541528 21103 consensus_queue.cc:237] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:36.546777 20798 catalog_manager.cc:5526] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 reported cstate change: term changed from 1 to 2, leader changed from 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130) to 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129). New cstate: current_term: 2 leader_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } health_report { overall_health: UNKNOWN } } }
I20250109 20:59:36.564869 21203 consensus_queue.cc:1035] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:59:36.567040 21203 consensus_queue.cc:1035] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:59:36.586412 21203 consensus_queue.cc:1035] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:59:36.794127 20976 raft_consensus.cc:1270] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Refusing update from remote peer 4dde147b5e4e4c20b5b3c20fedf7d576: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:59:36.795374 21103 consensus_queue.cc:1035] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:59:36.812999 21112 raft_consensus.cc:988] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272: : Instructing follower 3ba4ac9555ec4772814224be02daee66 to start an election
I20250109 20:59:36.813467 21203 raft_consensus.cc:1076] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Signalling peer 3ba4ac9555ec4772814224be02daee66 to start an election
I20250109 20:59:36.814765 21050 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583"
dest_uuid: "3ba4ac9555ec4772814224be02daee66"
from {username='slave'} at 127.0.0.1:38252
I20250109 20:59:36.815191 21050 raft_consensus.cc:491] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250109 20:59:36.815436 21050 raft_consensus.cc:3054] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:59:36.819695 21050 raft_consensus.cc:513] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:36.821147 21050 leader_election.cc:290] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [CANDIDATE]: Term 2 election: Requested vote from peers 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503), 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533)
I20250109 20:59:36.821833 20976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "3ba4ac9555ec4772814224be02daee66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "25d0989f1b044c549bec7fb6f03ca272"
I20250109 20:59:36.822079 20900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "3ba4ac9555ec4772814224be02daee66" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576"
I20250109 20:59:36.822413 20976 raft_consensus.cc:3049] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Stepping down as leader of term 1
I20250109 20:59:36.822587 20900 raft_consensus.cc:3054] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:59:36.822737 20976 raft_consensus.cc:738] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 25d0989f1b044c549bec7fb6f03ca272, State: Running, Role: LEADER
I20250109 20:59:36.823736 20976 consensus_queue.cc:260] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:36.824777 20976 raft_consensus.cc:3054] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:59:36.827306 20900 raft_consensus.cc:2463] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba4ac9555ec4772814224be02daee66 in term 2.
I20250109 20:59:36.828173 21015 leader_election.cc:304] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba4ac9555ec4772814224be02daee66, 4dde147b5e4e4c20b5b3c20fedf7d576; no voters:
I20250109 20:59:36.828914 21100 raft_consensus.cc:2798] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 20:59:36.829383 20976 raft_consensus.cc:2463] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba4ac9555ec4772814224be02daee66 in term 2.
I20250109 20:59:36.829908 21100 raft_consensus.cc:695] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 2 LEADER]: Becoming Leader. State: Replica: 3ba4ac9555ec4772814224be02daee66, State: Running, Role: LEADER
I20250109 20:59:36.830648 21100 consensus_queue.cc:237] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } }
I20250109 20:59:36.835602 20798 catalog_manager.cc:5526] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 reported cstate change: term changed from 1 to 2, leader changed from 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130) to 3ba4ac9555ec4772814224be02daee66 (127.17.14.131). New cstate: current_term: 2 leader_uuid: "3ba4ac9555ec4772814224be02daee66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } health_report { overall_health: UNKNOWN } } }
I20250109 20:59:36.862000 21050 raft_consensus.cc:1270] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Refusing update from remote peer 4dde147b5e4e4c20b5b3c20fedf7d576: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:59:36.863077 21103 consensus_queue.cc:1035] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:59:37.125000 20900 raft_consensus.cc:1270] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Refusing update from remote peer 3ba4ac9555ec4772814224be02daee66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:59:37.126232 21211 consensus_queue.cc:1035] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:59:37.128474 20976 raft_consensus.cc:1270] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Refusing update from remote peer 3ba4ac9555ec4772814224be02daee66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:59:37.129721 21211 consensus_queue.cc:1035] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:59:37.363669 21192 heartbeater.cc:502] Master 127.17.14.190:46305 was elected leader, sending a full tablet report...
I20250109 20:59:37.464437 21050 consensus_queue.cc:237] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } }
I20250109 20:59:37.470088 20900 raft_consensus.cc:1270] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Refusing update from remote peer 3ba4ac9555ec4772814224be02daee66: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 20:59:37.470431 20976 raft_consensus.cc:1270] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Refusing update from remote peer 3ba4ac9555ec4772814224be02daee66: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 20:59:37.471612 21211 consensus_queue.cc:1035] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 20:59:37.472496 21100 consensus_queue.cc:1035] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250109 20:59:37.480188 21013 consensus_peers.cc:487] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 -> Peer 3eede40835234fef911414d9c515e30a (127.17.14.132:43475): Couldn't send request to peer 3eede40835234fef911414d9c515e30a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65c1b0368bbb4c9eb5768d4412894583. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:59:37.480934 20900 raft_consensus.cc:2949] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3eede40835234fef911414d9c515e30a (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } } }
I20250109 20:59:37.481261 20976 raft_consensus.cc:2949] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3eede40835234fef911414d9c515e30a (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } } }
I20250109 20:59:37.493145 20799 catalog_manager.cc:5526] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 reported cstate change: config changed from index -1 to 3, NON_VOTER 3eede40835234fef911414d9c515e30a (127.17.14.132) added. New cstate: current_term: 2 leader_uuid: "3ba4ac9555ec4772814224be02daee66" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } } }
I20250109 20:59:37.560123 21100 raft_consensus.cc:2949] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3eede40835234fef911414d9c515e30a (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } } }
I20250109 20:59:37.570065 20976 consensus_queue.cc:237] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } }
I20250109 20:59:37.575250 20900 raft_consensus.cc:1270] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Refusing update from remote peer 25d0989f1b044c549bec7fb6f03ca272: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:59:37.575335 21050 raft_consensus.cc:1270] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Refusing update from remote peer 25d0989f1b044c549bec7fb6f03ca272: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:59:37.577069 21112 consensus_queue.cc:1035] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 20:59:37.577944 21102 consensus_queue.cc:1035] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250109 20:59:37.585984 20939 consensus_peers.cc:487] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 -> Peer 3eede40835234fef911414d9c515e30a (127.17.14.132:43475): Couldn't send request to peer 3eede40835234fef911414d9c515e30a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 060814a37ffc4435a40559089a4c4172. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:59:37.586297 21050 raft_consensus.cc:2949] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3eede40835234fef911414d9c515e30a (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } } }
I20250109 20:59:37.587570 20900 raft_consensus.cc:2949] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3eede40835234fef911414d9c515e30a (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } } }
I20250109 20:59:37.676139 20796 catalog_manager.cc:5526] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 reported cstate change: config changed from index -1 to 2, NON_VOTER 3eede40835234fef911414d9c515e30a (127.17.14.132) added. New cstate: current_term: 1 leader_uuid: "25d0989f1b044c549bec7fb6f03ca272" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } } }
I20250109 20:59:37.743989 21223 ts_tablet_manager.cc:927] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a: Initiating tablet copy from peer 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:37.745288 21223 tablet_copy_client.cc:323] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.131:35083
I20250109 20:59:37.753899 21060 tablet_copy_service.cc:140] P 3ba4ac9555ec4772814224be02daee66: Received BeginTabletCopySession request for tablet 65c1b0368bbb4c9eb5768d4412894583 from peer 3eede40835234fef911414d9c515e30a ({username='slave'} at 127.0.0.1:38286)
I20250109 20:59:37.754330 21060 tablet_copy_service.cc:161] P 3ba4ac9555ec4772814224be02daee66: Beginning new tablet copy session on tablet 65c1b0368bbb4c9eb5768d4412894583 from peer 3eede40835234fef911414d9c515e30a at {username='slave'} at 127.0.0.1:38286: session id = 3eede40835234fef911414d9c515e30a-65c1b0368bbb4c9eb5768d4412894583
I20250109 20:59:37.759548 21060 tablet_copy_source_session.cc:215] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 20:59:37.762333 21223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c1b0368bbb4c9eb5768d4412894583. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:37.771829 21223 tablet_copy_client.cc:806] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a: tablet copy: Starting download of 0 data blocks...
I20250109 20:59:37.772357 21223 tablet_copy_client.cc:670] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a: tablet copy: Starting download of 1 WAL segments...
I20250109 20:59:37.775089 21223 tablet_copy_client.cc:538] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 20:59:37.780277 21223 tablet_bootstrap.cc:492] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a: Bootstrap starting.
I20250109 20:59:37.788020 21112 raft_consensus.cc:2949] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3eede40835234fef911414d9c515e30a (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } } }
I20250109 20:59:37.795841 21223 tablet_bootstrap.cc:492] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:59:37.796770 21223 tablet_bootstrap.cc:492] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a: Bootstrap complete.
I20250109 20:59:37.797349 21223 ts_tablet_manager.cc:1397] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a: Time spent bootstrapping tablet: real 0.017s user 0.017s sys 0.001s
I20250109 20:59:37.799465 21223 raft_consensus.cc:357] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } }
I20250109 20:59:37.800204 21223 raft_consensus.cc:738] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3eede40835234fef911414d9c515e30a, State: Initialized, Role: LEARNER
I20250109 20:59:37.800830 21223 consensus_queue.cc:260] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } }
I20250109 20:59:37.803640 21223 ts_tablet_manager.cc:1428] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250109 20:59:37.805500 21060 tablet_copy_service.cc:342] P 3ba4ac9555ec4772814224be02daee66: Request end of tablet copy session 3eede40835234fef911414d9c515e30a-65c1b0368bbb4c9eb5768d4412894583 received from {username='slave'} at 127.0.0.1:38286
I20250109 20:59:37.805930 21060 tablet_copy_service.cc:434] P 3ba4ac9555ec4772814224be02daee66: ending tablet copy session 3eede40835234fef911414d9c515e30a-65c1b0368bbb4c9eb5768d4412894583 on tablet 65c1b0368bbb4c9eb5768d4412894583 with peer 3eede40835234fef911414d9c515e30a
I20250109 20:59:37.922983 21223 ts_tablet_manager.cc:927] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a: Initiating tablet copy from peer 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503)
I20250109 20:59:37.924575 21223 tablet_copy_client.cc:323] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.130:34503
I20250109 20:59:37.934129 20986 tablet_copy_service.cc:140] P 25d0989f1b044c549bec7fb6f03ca272: Received BeginTabletCopySession request for tablet 060814a37ffc4435a40559089a4c4172 from peer 3eede40835234fef911414d9c515e30a ({username='slave'} at 127.0.0.1:52706)
I20250109 20:59:37.934721 20986 tablet_copy_service.cc:161] P 25d0989f1b044c549bec7fb6f03ca272: Beginning new tablet copy session on tablet 060814a37ffc4435a40559089a4c4172 from peer 3eede40835234fef911414d9c515e30a at {username='slave'} at 127.0.0.1:52706: session id = 3eede40835234fef911414d9c515e30a-060814a37ffc4435a40559089a4c4172
I20250109 20:59:37.941668 20986 tablet_copy_source_session.cc:215] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 20:59:37.944321 21223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 060814a37ffc4435a40559089a4c4172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:37.957253 21223 tablet_copy_client.cc:806] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a: tablet copy: Starting download of 0 data blocks...
I20250109 20:59:37.957859 21223 tablet_copy_client.cc:670] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a: tablet copy: Starting download of 1 WAL segments...
I20250109 20:59:37.960713 21223 tablet_copy_client.cc:538] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 20:59:37.969272 21223 tablet_bootstrap.cc:492] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a: Bootstrap starting.
I20250109 20:59:37.979174 21167 raft_consensus.cc:1212] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250109 20:59:37.987519 21223 tablet_bootstrap.cc:492] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:59:37.988380 21223 tablet_bootstrap.cc:492] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a: Bootstrap complete.
I20250109 20:59:37.989019 21223 ts_tablet_manager.cc:1397] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.004s
I20250109 20:59:37.991318 21223 raft_consensus.cc:357] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } }
I20250109 20:59:37.992030 21223 raft_consensus.cc:738] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3eede40835234fef911414d9c515e30a, State: Initialized, Role: LEARNER
I20250109 20:59:37.992691 21223 consensus_queue.cc:260] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: true } }
I20250109 20:59:37.994984 21223 ts_tablet_manager.cc:1428] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20250109 20:59:37.996341 20986 tablet_copy_service.cc:342] P 25d0989f1b044c549bec7fb6f03ca272: Request end of tablet copy session 3eede40835234fef911414d9c515e30a-060814a37ffc4435a40559089a4c4172 received from {username='slave'} at 127.0.0.1:52706
I20250109 20:59:37.996640 20986 tablet_copy_service.cc:434] P 25d0989f1b044c549bec7fb6f03ca272: ending tablet copy session 3eede40835234fef911414d9c515e30a-060814a37ffc4435a40559089a4c4172 on tablet 060814a37ffc4435a40559089a4c4172 with peer 3eede40835234fef911414d9c515e30a
I20250109 20:59:38.252995 21167 raft_consensus.cc:1212] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250109 20:59:38.443298 20852 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:38.446465 20852 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:38.446849 20852 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:38.562618 21118 raft_consensus.cc:1059] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272: attempting to promote NON_VOTER 3eede40835234fef911414d9c515e30a to VOTER
I20250109 20:59:38.564569 21118 consensus_queue.cc:237] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } }
I20250109 20:59:38.570099 21049 raft_consensus.cc:1270] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Refusing update from remote peer 25d0989f1b044c549bec7fb6f03ca272: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 20:59:38.570436 21167 raft_consensus.cc:1270] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 1 LEARNER]: Refusing update from remote peer 25d0989f1b044c549bec7fb6f03ca272: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 20:59:38.570839 20900 raft_consensus.cc:1270] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Refusing update from remote peer 25d0989f1b044c549bec7fb6f03ca272: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 20:59:38.571691 21112 consensus_queue.cc:1035] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 20:59:38.572618 21118 consensus_queue.cc:1035] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 20:59:38.573412 21231 consensus_queue.cc:1035] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 20:59:38.584789 20900 raft_consensus.cc:2949] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3eede40835234fef911414d9c515e30a (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:38.587525 21100 raft_consensus.cc:1059] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66: attempting to promote NON_VOTER 3eede40835234fef911414d9c515e30a to VOTER
I20250109 20:59:38.587138 21167 raft_consensus.cc:2949] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3eede40835234fef911414d9c515e30a (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:38.589287 21100 consensus_queue.cc:237] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } }
I20250109 20:59:38.598150 20900 raft_consensus.cc:1270] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Refusing update from remote peer 3ba4ac9555ec4772814224be02daee66: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 20:59:38.598258 21166 raft_consensus.cc:1270] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [term 2 LEARNER]: Refusing update from remote peer 3ba4ac9555ec4772814224be02daee66: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 20:59:38.600342 20799 catalog_manager.cc:5526] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 reported cstate change: config changed from index 2 to 3, 3eede40835234fef911414d9c515e30a (127.17.14.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "25d0989f1b044c549bec7fb6f03ca272" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:38.604379 20975 raft_consensus.cc:1270] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Refusing update from remote peer 3ba4ac9555ec4772814224be02daee66: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 20:59:38.604914 21213 consensus_queue.cc:1035] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 20:59:38.606817 21213 consensus_queue.cc:1035] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 20:59:38.610477 21234 consensus_queue.cc:1035] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20250109 20:59:38.625416 21167 raft_consensus.cc:2949] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3eede40835234fef911414d9c515e30a (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:38.637267 20975 raft_consensus.cc:2949] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3eede40835234fef911414d9c515e30a (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:38.647673 20796 catalog_manager.cc:5526] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 reported cstate change: config changed from index 3 to 4, 3eede40835234fef911414d9c515e30a (127.17.14.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3ba4ac9555ec4772814224be02daee66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:38.848981 21112 raft_consensus.cc:2949] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3eede40835234fef911414d9c515e30a (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:38.857471 21049 raft_consensus.cc:2949] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3eede40835234fef911414d9c515e30a (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } attrs { replace: true } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:38.882612 21234 raft_consensus.cc:1022] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 2 LEADER]: attempt to promote peer 3eede40835234fef911414d9c515e30a: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250109 20:59:38.895843 20900 raft_consensus.cc:2949] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3eede40835234fef911414d9c515e30a (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:39.027081 20975 consensus_queue.cc:237] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } }
I20250109 20:59:39.031019 21167 raft_consensus.cc:1270] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 1 FOLLOWER]: Refusing update from remote peer 25d0989f1b044c549bec7fb6f03ca272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250109 20:59:39.031162 21050 raft_consensus.cc:1270] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Refusing update from remote peer 25d0989f1b044c549bec7fb6f03ca272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250109 20:59:39.032517 21241 consensus_queue.cc:1035] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 20:59:39.033232 21118 consensus_queue.cc:1035] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 20:59:39.040876 21167 raft_consensus.cc:2949] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:39.041189 21050 raft_consensus.cc:2949] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:39.051807 20799 catalog_manager.cc:5526] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 reported cstate change: config changed from index 3 to 4, VOTER 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129) evicted. New cstate: current_term: 1 leader_uuid: "25d0989f1b044c549bec7fb6f03ca272" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:39.056885 21211 raft_consensus.cc:2949] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3eede40835234fef911414d9c515e30a (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } attrs { replace: true } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:39.089358 20889 tablet_service.cc:1514] Processing DeleteTablet for tablet 060814a37ffc4435a40559089a4c4172 with delete_type TABLET_DATA_TOMBSTONED (TS 4dde147b5e4e4c20b5b3c20fedf7d576 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59688
I20250109 20:59:39.091804 21242 tablet_replica.cc:331] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576: stopping tablet replica
I20250109 20:59:39.093029 21242 raft_consensus.cc:2238] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:59:39.093773 21242 raft_consensus.cc:2267] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:39.097326 21242 ts_tablet_manager.cc:1905] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250109 20:59:39.099694 21049 consensus_queue.cc:237] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } }
I20250109 20:59:39.105113 21167 raft_consensus.cc:1270] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Refusing update from remote peer 3ba4ac9555ec4772814224be02daee66: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 20:59:39.106647 21100 consensus_queue.cc:1035] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:59:39.107293 20900 raft_consensus.cc:1270] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Refusing update from remote peer 3ba4ac9555ec4772814224be02daee66: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 20:59:39.108637 21100 consensus_queue.cc:1035] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:59:39.115272 21242 ts_tablet_manager.cc:1918] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250109 20:59:39.115605 21242 log.cc:1198] T 060814a37ffc4435a40559089a4c4172 P 4dde147b5e4e4c20b5b3c20fedf7d576: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-0-root/wals/060814a37ffc4435a40559089a4c4172
I20250109 20:59:39.116981 20785 catalog_manager.cc:4872] TS 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533): tablet 060814a37ffc4435a40559089a4c4172 (table test-workload [id=72ff5ee9e9764c7ead5d3d2eaeb14842]) successfully deleted
I20250109 20:59:39.116889 20900 raft_consensus.cc:2949] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:39.114784 21167 raft_consensus.cc:2949] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:39.128399 20798 catalog_manager.cc:5526] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 reported cstate change: config changed from index 4 to 5, VOTER 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130) evicted. New cstate: current_term: 2 leader_uuid: "3ba4ac9555ec4772814224be02daee66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:39.315773 20964 tablet_service.cc:1514] Processing DeleteTablet for tablet 65c1b0368bbb4c9eb5768d4412894583 with delete_type TABLET_DATA_TOMBSTONED (TS 25d0989f1b044c549bec7fb6f03ca272 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52650
I20250109 20:59:39.318739 21243 tablet_replica.cc:331] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272: stopping tablet replica
I20250109 20:59:39.319710 21243 raft_consensus.cc:2238] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:59:39.320397 21243 raft_consensus.cc:2267] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:39.322841 21243 ts_tablet_manager.cc:1905] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250109 20:59:39.334411 21243 ts_tablet_manager.cc:1918] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250109 20:59:39.334744 21243 log.cc:1198] T 65c1b0368bbb4c9eb5768d4412894583 P 25d0989f1b044c549bec7fb6f03ca272: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736456270445890-17466-0/minicluster-data/ts-1-root/wals/65c1b0368bbb4c9eb5768d4412894583
I20250109 20:59:39.335836 20785 catalog_manager.cc:4872] TS 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503): tablet 65c1b0368bbb4c9eb5768d4412894583 (table test-workload [id=72ff5ee9e9764c7ead5d3d2eaeb14842]) successfully deleted
I20250109 20:59:39.397426 21112 raft_consensus.cc:2949] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:39.403934 20785 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 060814a37ffc4435a40559089a4c4172 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 20:59:39.661288 21100 raft_consensus.cc:2949] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } } }
I20250109 20:59:39.667613 20783 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 65c1b0368bbb4c9eb5768d4412894583 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 20:59:40.447808 20852 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:40.450290 20852 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:40.450637 20852 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:44.109726 21245 raft_consensus.cc:491] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dde147b5e4e4c20b5b3c20fedf7d576)
I20250109 20:59:44.111137 21245 raft_consensus.cc:513] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:44.115471 21245 leader_election.cc:290] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533), 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:44.300729 21248 raft_consensus.cc:491] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3ba4ac9555ec4772814224be02daee66)
I20250109 20:59:44.301530 21248 raft_consensus.cc:513] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } }
I20250109 20:59:44.305915 21248 leader_election.cc:290] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083), 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533)
I20250109 20:59:44.323102 21247 raft_consensus.cc:491] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25d0989f1b044c549bec7fb6f03ca272)
I20250109 20:59:44.323853 21247 raft_consensus.cc:513] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } }
I20250109 20:59:44.325639 21247 leader_election.cc:290] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503), 3eede40835234fef911414d9c515e30a (127.17.14.132:43475)
I20250109 20:59:44.327044 20975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060814a37ffc4435a40559089a4c4172" candidate_uuid: "3ba4ac9555ec4772814224be02daee66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "25d0989f1b044c549bec7fb6f03ca272" is_pre_election: true
I20250109 20:59:44.330406 21248 raft_consensus.cc:491] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25d0989f1b044c549bec7fb6f03ca272)
I20250109 20:59:44.331167 21248 raft_consensus.cc:513] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } }
I20250109 20:59:44.335114 20975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060814a37ffc4435a40559089a4c4172" candidate_uuid: "3eede40835234fef911414d9c515e30a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "25d0989f1b044c549bec7fb6f03ca272" is_pre_election: true
I20250109 20:59:44.335420 21248 leader_election.cc:290] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503), 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:44.347996 21049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66" is_pre_election: true
I20250109 20:59:44.348716 21049 raft_consensus.cc:2463] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 2.
I20250109 20:59:44.348627 21048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "3eede40835234fef911414d9c515e30a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66" is_pre_election: true
I20250109 20:59:44.350370 21047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060814a37ffc4435a40559089a4c4172" candidate_uuid: "3eede40835234fef911414d9c515e30a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66" is_pre_election: true
I20250109 20:59:44.350370 20939 leader_election.cc:304] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25d0989f1b044c549bec7fb6f03ca272, 3ba4ac9555ec4772814224be02daee66; no voters:
I20250109 20:59:44.351121 21047 raft_consensus.cc:2463] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eede40835234fef911414d9c515e30a in term 1.
I20250109 20:59:44.351919 21245 raft_consensus.cc:2798] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250109 20:59:44.352250 21245 raft_consensus.cc:491] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4dde147b5e4e4c20b5b3c20fedf7d576)
I20250109 20:59:44.352574 21245 raft_consensus.cc:3054] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Advancing to term 3
I20250109 20:59:44.355808 21130 leader_election.cc:304] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba4ac9555ec4772814224be02daee66, 3eede40835234fef911414d9c515e30a; no voters: 25d0989f1b044c549bec7fb6f03ca272
I20250109 20:59:44.358181 21248 raft_consensus.cc:2798] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250109 20:59:44.362053 21248 raft_consensus.cc:491] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 25d0989f1b044c549bec7fb6f03ca272)
I20250109 20:59:44.369666 21248 raft_consensus.cc:3054] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:59:44.367950 21250 raft_consensus.cc:491] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3ba4ac9555ec4772814224be02daee66)
I20250109 20:59:44.371377 21250 raft_consensus.cc:513] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } }
I20250109 20:59:44.374068 21250 leader_election.cc:290] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083), 3eede40835234fef911414d9c515e30a (127.17.14.132:43475)
I20250109 20:59:44.375674 21245 raft_consensus.cc:513] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:44.376893 20975 raft_consensus.cc:1235] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 3 FOLLOWER]: Rejecting Update request from peer 4dde147b5e4e4c20b5b3c20fedf7d576 for earlier term 2. Current term is 3. Ops: []
I20250109 20:59:44.377765 21245 leader_election.cc:290] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 3 election: Requested vote from peers 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533), 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:44.381814 21047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66" is_pre_election: true
I20250109 20:59:44.382421 21048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66"
I20250109 20:59:44.386821 21166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060814a37ffc4435a40559089a4c4172" candidate_uuid: "3ba4ac9555ec4772814224be02daee66" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eede40835234fef911414d9c515e30a" is_pre_election: true
I20250109 20:59:44.387495 20900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" is_pre_election: true
I20250109 20:59:44.388418 20901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576"
I20250109 20:59:44.391731 20941 leader_election.cc:304] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25d0989f1b044c549bec7fb6f03ca272; no voters: 3ba4ac9555ec4772814224be02daee66, 4dde147b5e4e4c20b5b3c20fedf7d576
I20250109 20:59:44.391965 21248 raft_consensus.cc:513] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } }
I20250109 20:59:44.393409 21166 raft_consensus.cc:2388] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3ba4ac9555ec4772814224be02daee66 in current term 2: Already voted for candidate 3eede40835234fef911414d9c515e30a in this term.
I20250109 20:59:44.394452 21245 raft_consensus.cc:2743] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250109 20:59:44.395102 21248 leader_election.cc:290] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [CANDIDATE]: Term 2 election: Requested vote from peers 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503), 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:44.395198 20976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060814a37ffc4435a40559089a4c4172" candidate_uuid: "3eede40835234fef911414d9c515e30a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "25d0989f1b044c549bec7fb6f03ca272"
I20250109 20:59:44.396247 21165 raft_consensus.cc:1235] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Rejecting Update request from peer 25d0989f1b044c549bec7fb6f03ca272 for earlier term 1. Current term is 2. Ops: []
I20250109 20:59:44.402757 21048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060814a37ffc4435a40559089a4c4172" candidate_uuid: "3eede40835234fef911414d9c515e30a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66"
I20250109 20:59:44.403259 21048 raft_consensus.cc:3054] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:59:44.408605 21013 leader_election.cc:304] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba4ac9555ec4772814224be02daee66; no voters: 25d0989f1b044c549bec7fb6f03ca272, 3eede40835234fef911414d9c515e30a
I20250109 20:59:44.409333 20852 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:44.413128 20852 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:44.420842 20901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "3eede40835234fef911414d9c515e30a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" is_pre_election: true
I20250109 20:59:44.422614 21132 leader_election.cc:304] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eede40835234fef911414d9c515e30a; no voters: 3ba4ac9555ec4772814224be02daee66, 4dde147b5e4e4c20b5b3c20fedf7d576
I20250109 20:59:44.422418 21251 consensus_queue.cc:1046] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250109 20:59:44.423661 21248 raft_consensus.cc:2743] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250109 20:59:44.426371 20852 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:44.426931 21245 consensus_queue.cc:1046] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:59:44.429279 21048 raft_consensus.cc:2463] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eede40835234fef911414d9c515e30a in term 2.
I20250109 20:59:44.430202 21050 raft_consensus.cc:1235] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Rejecting Update request from peer 25d0989f1b044c549bec7fb6f03ca272 for earlier term 1. Current term is 2. Ops: []
I20250109 20:59:44.430490 21130 leader_election.cc:304] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba4ac9555ec4772814224be02daee66, 3eede40835234fef911414d9c515e30a; no voters: 25d0989f1b044c549bec7fb6f03ca272
I20250109 20:59:44.432050 21248 raft_consensus.cc:2798] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Leader election won for term 2
I20250109 20:59:44.433957 21165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c1b0368bbb4c9eb5768d4412894583" candidate_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eede40835234fef911414d9c515e30a" is_pre_election: true
I20250109 20:59:44.435446 21246 consensus_queue.cc:1046] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250109 20:59:44.436839 21247 raft_consensus.cc:2743] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:59:44.436990 20864 leader_election.cc:304] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dde147b5e4e4c20b5b3c20fedf7d576; no voters: 3ba4ac9555ec4772814224be02daee66, 3eede40835234fef911414d9c515e30a
I20250109 20:59:44.437911 21251 raft_consensus.cc:2743] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250109 20:59:44.439086 21248 raft_consensus.cc:695] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 2 LEADER]: Becoming Leader. State: Replica: 3eede40835234fef911414d9c515e30a, State: Running, Role: LEADER
I20250109 20:59:44.440024 21248 consensus_queue.cc:237] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } }
I20250109 20:59:44.448699 20797 catalog_manager.cc:5526] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a reported cstate change: term changed from 1 to 2, leader changed from 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130) to 3eede40835234fef911414d9c515e30a (127.17.14.132). New cstate: current_term: 2 leader_uuid: "3eede40835234fef911414d9c515e30a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250109 20:59:44.465559 21050 raft_consensus.cc:1235] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Rejecting Update request from peer 25d0989f1b044c549bec7fb6f03ca272 for earlier term 1. Current term is 2. Ops: []
I20250109 20:59:44.466624 21246 consensus_queue.cc:1046] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:59:44.473927 21165 raft_consensus.cc:1235] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 2 LEADER]: Rejecting Update request from peer 25d0989f1b044c549bec7fb6f03ca272 for earlier term 1. Current term is 2. Ops: []
I20250109 20:59:44.474987 21246 consensus_queue.cc:1046] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:59:44.475541 21246 consensus_queue.cc:1225] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Peer 3eede40835234fef911414d9c515e30a log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250109 20:59:44.624703 20975 raft_consensus.cc:1235] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 3 FOLLOWER]: Rejecting Update request from peer 4dde147b5e4e4c20b5b3c20fedf7d576 for earlier term 2. Current term is 3. Ops: []
I20250109 20:59:44.626204 21251 consensus_queue.cc:1046] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 20:59:44.648197 21245 raft_consensus.cc:3049] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Stepping down as leader of term 1
I20250109 20:59:44.648553 21245 raft_consensus.cc:738] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 25d0989f1b044c549bec7fb6f03ca272, State: Running, Role: LEADER
I20250109 20:59:44.649367 21245 consensus_queue.cc:260] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } } peers { permanent_uuid: "3eede40835234fef911414d9c515e30a" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 43475 } attrs { promote: false } }
I20250109 20:59:44.650413 21245 raft_consensus.cc:3054] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:59:44.683068 20975 raft_consensus.cc:1270] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Refusing update from remote peer 3eede40835234fef911414d9c515e30a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 20:59:44.684367 21248 consensus_queue.cc:1035] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [LEADER]: Connected to new peer: Peer: permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:59:44.720917 21252 raft_consensus.cc:3049] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 LEADER]: Stepping down as leader of term 2
I20250109 20:59:44.721302 21252 raft_consensus.cc:738] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4dde147b5e4e4c20b5b3c20fedf7d576, State: Running, Role: LEADER
I20250109 20:59:44.722018 21252 consensus_queue.cc:260] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:44.723052 21252 raft_consensus.cc:3054] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Advancing to term 3
I20250109 20:59:44.751298 21050 raft_consensus.cc:1270] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Refusing update from remote peer 3eede40835234fef911414d9c515e30a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 20:59:44.752419 21248 consensus_queue.cc:1035] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:59:45.361527 21245 raft_consensus.cc:491] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:59:45.361945 21245 raft_consensus.cc:513] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:45.363500 21245 leader_election.cc:290] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533), 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:45.364291 20901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" is_pre_election: true
I20250109 20:59:45.364432 21050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66" is_pre_election: true
I20250109 20:59:45.364889 20901 raft_consensus.cc:2463] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 3.
I20250109 20:59:45.365741 20941 leader_election.cc:304] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25d0989f1b044c549bec7fb6f03ca272, 4dde147b5e4e4c20b5b3c20fedf7d576; no voters: 3ba4ac9555ec4772814224be02daee66
I20250109 20:59:45.366335 21245 raft_consensus.cc:2798] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250109 20:59:45.366513 21188 debug-util.cc:398] Leaking SignalData structure 0x7b080018dbc0 after lost signal to thread 17469
I20250109 20:59:45.366642 21245 raft_consensus.cc:491] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:59:45.367063 21245 raft_consensus.cc:3054] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 3 FOLLOWER]: Advancing to term 4
W20250109 20:59:45.367511 21188 debug-util.cc:398] Leaking SignalData structure 0x7b08001f9400 after lost signal to thread 20833
W20250109 20:59:45.369149 21188 debug-util.cc:398] Leaking SignalData structure 0x7b0800234b80 after lost signal to thread 21000
W20250109 20:59:45.370004 21188 debug-util.cc:398] Leaking SignalData structure 0x7b080015e380 after lost signal to thread 21074
I20250109 20:59:45.371779 21245 raft_consensus.cc:513] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:45.373274 21245 leader_election.cc:290] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 4 election: Requested vote from peers 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533), 3ba4ac9555ec4772814224be02daee66 (127.17.14.131:35083)
I20250109 20:59:45.374239 20901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576"
I20250109 20:59:45.374303 21050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "25d0989f1b044c549bec7fb6f03ca272" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3ba4ac9555ec4772814224be02daee66"
I20250109 20:59:45.374784 20901 raft_consensus.cc:3054] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 3 FOLLOWER]: Advancing to term 4
I20250109 20:59:45.379348 20901 raft_consensus.cc:2463] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25d0989f1b044c549bec7fb6f03ca272 in term 4.
I20250109 20:59:45.380138 20941 leader_election.cc:304] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25d0989f1b044c549bec7fb6f03ca272, 4dde147b5e4e4c20b5b3c20fedf7d576; no voters: 3ba4ac9555ec4772814224be02daee66
I20250109 20:59:45.380729 21245 raft_consensus.cc:2798] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 4 FOLLOWER]: Leader election won for term 4
I20250109 20:59:45.381770 21245 raft_consensus.cc:695] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 4 LEADER]: Becoming Leader. State: Replica: 25d0989f1b044c549bec7fb6f03ca272, State: Running, Role: LEADER
I20250109 20:59:45.382732 21245 consensus_queue.cc:237] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:45.390120 20797 catalog_manager.cc:5526] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 reported cstate change: term changed from 2 to 4, leader changed from 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129) to 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130). New cstate: current_term: 4 leader_uuid: "25d0989f1b044c549bec7fb6f03ca272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } health_report { overall_health: UNKNOWN } } }
I20250109 20:59:45.543581 21247 raft_consensus.cc:491] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dde147b5e4e4c20b5b3c20fedf7d576)
I20250109 20:59:45.544179 21247 raft_consensus.cc:513] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 } } peers { permanent_uuid: "25d0989f1b044c549bec7fb6f03ca272" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 34503 } } peers { permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 } }
I20250109 20:59:45.546352 21247 leader_election.cc:290] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533), 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503)
I20250109 20:59:45.547626 20901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "3ba4ac9555ec4772814224be02daee66" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" is_pre_election: true
I20250109 20:59:45.548228 20901 raft_consensus.cc:2363] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3ba4ac9555ec4772814224be02daee66 for earlier term 3. Current term is 4.
I20250109 20:59:45.549408 21015 leader_election.cc:400] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533) with higher term. Message: Invalid argument: T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3ba4ac9555ec4772814224be02daee66 for earlier term 3. Current term is 4.
I20250109 20:59:45.549702 21015 leader_election.cc:403] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250109 20:59:45.549763 20975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5dcf8c0f9e74e39ba0bd00de847ef6d" candidate_uuid: "3ba4ac9555ec4772814224be02daee66" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "25d0989f1b044c549bec7fb6f03ca272" is_pre_election: true
I20250109 20:59:45.550408 21247 raft_consensus.cc:3054] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Advancing to term 4
I20250109 20:59:45.558073 21247 raft_consensus.cc:2743] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533) with higher term. Message: Invalid argument: T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3ba4ac9555ec4772814224be02daee66 for earlier term 3. Current term is 4.
I20250109 20:59:45.698763 21050 raft_consensus.cc:1270] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 4 FOLLOWER]: Refusing update from remote peer 25d0989f1b044c549bec7fb6f03ca272: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250109 20:59:45.699908 21245 consensus_queue.cc:1035] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba4ac9555ec4772814224be02daee66" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 35083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 20:59:45.717124 20901 raft_consensus.cc:1270] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 4 FOLLOWER]: Refusing update from remote peer 25d0989f1b044c549bec7fb6f03ca272: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250109 20:59:45.718313 21245 consensus_queue.cc:1035] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dde147b5e4e4c20b5b3c20fedf7d576" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 38533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 20:59:46.427511 20852 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:46.431134 20852 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:46.431597 20852 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:48.432528 20852 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:48.434960 20852 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:48.435259 20852 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:48.581650 17466 tablet_server.cc:178] TabletServer@127.17.14.129:0 shutting down...
I20250109 20:59:48.601732 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:48.602432 17466 tablet_replica.cc:331] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576: stopping tablet replica
I20250109 20:59:48.603145 17466 raft_consensus.cc:2238] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250109 20:59:48.603538 17466 raft_consensus.cc:2267] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:48.605461 17466 tablet_replica.cc:331] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576: stopping tablet replica
I20250109 20:59:48.605974 17466 raft_consensus.cc:2238] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:59:48.606984 17466 raft_consensus.cc:2267] T 65c1b0368bbb4c9eb5768d4412894583 P 4dde147b5e4e4c20b5b3c20fedf7d576 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:48.630638 17466 tablet_server.cc:195] TabletServer@127.17.14.129:0 shutdown complete.
I20250109 20:59:48.645368 17466 tablet_server.cc:178] TabletServer@127.17.14.130:0 shutting down...
I20250109 20:59:48.664630 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:48.665248 17466 tablet_replica.cc:331] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272: stopping tablet replica
I20250109 20:59:48.665820 17466 raft_consensus.cc:2238] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:59:48.666191 17466 raft_consensus.cc:2267] T 060814a37ffc4435a40559089a4c4172 P 25d0989f1b044c549bec7fb6f03ca272 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:48.668107 17466 tablet_replica.cc:331] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272: stopping tablet replica
I20250109 20:59:48.668534 17466 raft_consensus.cc:2238] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 4 LEADER]: Raft consensus shutting down.
I20250109 20:59:48.669193 17466 raft_consensus.cc:2267] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 25d0989f1b044c549bec7fb6f03ca272 [term 4 FOLLOWER]: Raft consensus is shut down!
W20250109 20:59:48.678480 21015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.14.129:38533: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20250109 20:59:48.682467 21015 consensus_peers.cc:487] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 -> Peer 4dde147b5e4e4c20b5b3c20fedf7d576 (127.17.14.129:38533): Couldn't send request to peer 4dde147b5e4e4c20b5b3c20fedf7d576. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:38533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 20:59:48.693053 17466 tablet_server.cc:195] TabletServer@127.17.14.130:0 shutdown complete.
I20250109 20:59:48.705745 17466 tablet_server.cc:178] TabletServer@127.17.14.131:0 shutting down...
W20250109 20:59:48.725076 21132 consensus_peers.cc:487] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a -> Peer 25d0989f1b044c549bec7fb6f03ca272 (127.17.14.130:34503): Couldn't send request to peer 25d0989f1b044c549bec7fb6f03ca272. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.130:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 20:59:48.732439 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:48.733130 17466 tablet_replica.cc:331] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66: stopping tablet replica
I20250109 20:59:48.733793 17466 raft_consensus.cc:2238] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250109 20:59:48.734200 17466 raft_consensus.cc:2267] T f5dcf8c0f9e74e39ba0bd00de847ef6d P 3ba4ac9555ec4772814224be02daee66 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:48.735945 17466 tablet_replica.cc:331] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66: stopping tablet replica
I20250109 20:59:48.736347 17466 raft_consensus.cc:2238] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:59:48.736685 17466 raft_consensus.cc:2267] T 060814a37ffc4435a40559089a4c4172 P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:48.738356 17466 tablet_replica.cc:331] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66: stopping tablet replica
I20250109 20:59:48.738778 17466 raft_consensus.cc:2238] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 2 LEADER]: Raft consensus shutting down.
I20250109 20:59:48.739465 17466 raft_consensus.cc:2267] T 65c1b0368bbb4c9eb5768d4412894583 P 3ba4ac9555ec4772814224be02daee66 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:48.761652 17466 tablet_server.cc:195] TabletServer@127.17.14.131:0 shutdown complete.
I20250109 20:59:48.773999 17466 tablet_server.cc:178] TabletServer@127.17.14.132:0 shutting down...
I20250109 20:59:48.792716 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:48.793293 17466 tablet_replica.cc:331] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a: stopping tablet replica
I20250109 20:59:48.793859 17466 raft_consensus.cc:2238] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 2 LEADER]: Raft consensus shutting down.
I20250109 20:59:48.794566 17466 raft_consensus.cc:2267] T 060814a37ffc4435a40559089a4c4172 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:48.796375 17466 tablet_replica.cc:331] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a: stopping tablet replica
I20250109 20:59:48.796797 17466 raft_consensus.cc:2238] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 20:59:48.797317 17466 raft_consensus.cc:2267] T 65c1b0368bbb4c9eb5768d4412894583 P 3eede40835234fef911414d9c515e30a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:48.833149 17466 tablet_server.cc:195] TabletServer@127.17.14.132:0 shutdown complete.
I20250109 20:59:48.846402 17466 master.cc:537] Master@127.17.14.190:46305 shutting down...
I20250109 20:59:48.862578 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:59:48.863055 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:48.863363 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P fd1179d89ac5492bbdf06b42cc3a5553: stopping tablet replica
I20250109 20:59:48.881398 17466 master.cc:559] Master@127.17.14.190:46305 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (14617 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20250109 20:59:48.918154 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:36697
I20250109 20:59:48.919106 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:48.925243 21286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:48.926218 21287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:48.927438 21289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:48.928452 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:48.929148 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:48.929335 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:48.929441 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456388929431 us; error 0 us; skew 500 ppm
I20250109 20:59:48.929965 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:48.936949 17466 webserver.cc:458] Webserver started at http://127.17.14.190:36575/ using document root <none> and password file <none>
I20250109 20:59:48.937407 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:48.937616 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:48.937875 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:48.938992 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "ede37d3c23d4484d95f5d93714530ec1"
format_stamp: "Formatted at 2025-01-09 20:59:48 on dist-test-slave-j54z"
I20250109 20:59:48.943159 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250109 20:59:48.946050 21294 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:48.946696 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:59:48.946936 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "ede37d3c23d4484d95f5d93714530ec1"
format_stamp: "Formatted at 2025-01-09 20:59:48 on dist-test-slave-j54z"
I20250109 20:59:48.947180 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:48.956116 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:48.957185 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:48.992813 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:36697
I20250109 20:59:48.992890 21345 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:36697 every 8 connection(s)
I20250109 20:59:48.996575 21346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:49.006273 21346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1: Bootstrap starting.
I20250109 20:59:49.010556 21346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:49.014335 21346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1: No bootstrap required, opened a new log
I20250109 20:59:49.016203 21346 raft_consensus.cc:357] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ede37d3c23d4484d95f5d93714530ec1" member_type: VOTER }
I20250109 20:59:49.016546 21346 raft_consensus.cc:383] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:49.016769 21346 raft_consensus.cc:738] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede37d3c23d4484d95f5d93714530ec1, State: Initialized, Role: FOLLOWER
I20250109 20:59:49.017294 21346 consensus_queue.cc:260] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ede37d3c23d4484d95f5d93714530ec1" member_type: VOTER }
I20250109 20:59:49.017755 21346 raft_consensus.cc:397] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 20:59:49.017961 21346 raft_consensus.cc:491] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 20:59:49.018196 21346 raft_consensus.cc:3054] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:49.022203 21346 raft_consensus.cc:513] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ede37d3c23d4484d95f5d93714530ec1" member_type: VOTER }
I20250109 20:59:49.022655 21346 leader_election.cc:304] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ede37d3c23d4484d95f5d93714530ec1; no voters:
I20250109 20:59:49.023669 21346 leader_election.cc:290] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 20:59:49.024024 21349 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:59:49.025267 21349 raft_consensus.cc:695] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [term 1 LEADER]: Becoming Leader. State: Replica: ede37d3c23d4484d95f5d93714530ec1, State: Running, Role: LEADER
I20250109 20:59:49.025816 21349 consensus_queue.cc:237] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ede37d3c23d4484d95f5d93714530ec1" member_type: VOTER }
I20250109 20:59:49.026506 21346 sys_catalog.cc:564] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:59:49.029965 21350 sys_catalog.cc:455] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ede37d3c23d4484d95f5d93714530ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ede37d3c23d4484d95f5d93714530ec1" member_type: VOTER } }
I20250109 20:59:49.030661 21350 sys_catalog.cc:458] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [sys.catalog]: This master's current role is: LEADER
I20250109 20:59:49.030606 21351 sys_catalog.cc:455] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ede37d3c23d4484d95f5d93714530ec1. Latest consensus state: current_term: 1 leader_uuid: "ede37d3c23d4484d95f5d93714530ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ede37d3c23d4484d95f5d93714530ec1" member_type: VOTER } }
I20250109 20:59:49.031221 21351 sys_catalog.cc:458] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [sys.catalog]: This master's current role is: LEADER
I20250109 20:59:49.032681 21356 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:59:49.039506 21356 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:59:49.040395 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 20:59:49.047540 21356 catalog_manager.cc:1348] Generated new cluster ID: 018eac646e3f40749f9860e99b6fe29b
I20250109 20:59:49.047758 21356 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:59:49.087147 21356 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 20:59:49.088274 21356 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:59:49.106325 21356 catalog_manager.cc:5899] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1: Generated new TSK 0
I20250109 20:59:49.106830 21356 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:59:49.171170 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:49.177670 21367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:49.178915 21368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:49.179860 21370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:49.182102 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:49.182911 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:49.183127 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:49.183276 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456389183260 us; error 0 us; skew 500 ppm
I20250109 20:59:49.183740 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:49.186309 17466 webserver.cc:458] Webserver started at http://127.17.14.129:41257/ using document root <none> and password file <none>
I20250109 20:59:49.186741 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:49.186911 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:49.187150 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:49.188261 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-0-root/instance:
uuid: "981558576c57415fae830c97bf70581d"
format_stamp: "Formatted at 2025-01-09 20:59:49 on dist-test-slave-j54z"
I20250109 20:59:49.192401 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 20:59:49.195421 21375 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:49.196061 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 20:59:49.196323 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "981558576c57415fae830c97bf70581d"
format_stamp: "Formatted at 2025-01-09 20:59:49 on dist-test-slave-j54z"
I20250109 20:59:49.196574 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:49.206528 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:49.207576 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:49.208875 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:49.211288 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:49.211476 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:49.211687 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:49.211843 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:49.249097 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:41401
I20250109 20:59:49.249253 21437 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:41401 every 8 connection(s)
I20250109 20:59:49.253551 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:49.262111 21442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:49.262526 21443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:49.264751 21445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:49.266238 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:49.266752 21438 heartbeater.cc:346] Connected to a master server at 127.17.14.190:36697
I20250109 20:59:49.267143 21438 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:49.267225 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:49.267503 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:49.267686 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456389267669 us; error 0 us; skew 500 ppm
I20250109 20:59:49.268004 21438 heartbeater.cc:510] Master 127.17.14.190:36697 requested a full tablet report, sending...
I20250109 20:59:49.268368 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:49.270308 21311 ts_manager.cc:194] Registered new tserver with Master: 981558576c57415fae830c97bf70581d (127.17.14.129:41401)
I20250109 20:59:49.271093 17466 webserver.cc:458] Webserver started at http://127.17.14.130:39851/ using document root <none> and password file <none>
I20250109 20:59:49.271533 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:49.271699 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:49.271939 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:49.272181 21311 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50632
I20250109 20:59:49.273010 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-1-root/instance:
uuid: "549513f3278c46feba3073887daecfce"
format_stamp: "Formatted at 2025-01-09 20:59:49 on dist-test-slave-j54z"
I20250109 20:59:49.277328 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20250109 20:59:49.280284 21450 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:49.280920 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 20:59:49.281168 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "549513f3278c46feba3073887daecfce"
format_stamp: "Formatted at 2025-01-09 20:59:49 on dist-test-slave-j54z"
I20250109 20:59:49.281430 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:49.298048 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:49.299135 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:49.300464 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:49.302608 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:49.302800 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:49.303007 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:49.303153 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:49.340701 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:42503
I20250109 20:59:49.340793 21512 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:42503 every 8 connection(s)
I20250109 20:59:49.345166 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:59:49.352633 21516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:49.353638 21517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:59:49.355340 21519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:59:49.359097 21513 heartbeater.cc:346] Connected to a master server at 127.17.14.190:36697
I20250109 20:59:49.359652 21513 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:49.360724 21513 heartbeater.cc:510] Master 127.17.14.190:36697 requested a full tablet report, sending...
I20250109 20:59:49.363003 21311 ts_manager.cc:194] Registered new tserver with Master: 549513f3278c46feba3073887daecfce (127.17.14.130:42503)
I20250109 20:59:49.363827 17466 server_base.cc:1034] running on GCE node
I20250109 20:59:49.364588 21311 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50642
I20250109 20:59:49.364858 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 20:59:49.365111 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 20:59:49.365304 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456389365289 us; error 0 us; skew 500 ppm
I20250109 20:59:49.365965 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:59:49.368182 17466 webserver.cc:458] Webserver started at http://127.17.14.131:36781/ using document root <none> and password file <none>
I20250109 20:59:49.368624 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 20:59:49.368798 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:59:49.369030 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:59:49.370146 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-2-root/instance:
uuid: "f7deb6d8db534064a603046070039e3a"
format_stamp: "Formatted at 2025-01-09 20:59:49 on dist-test-slave-j54z"
I20250109 20:59:49.374301 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20250109 20:59:49.377177 21524 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:49.377904 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250109 20:59:49.378168 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "f7deb6d8db534064a603046070039e3a"
format_stamp: "Formatted at 2025-01-09 20:59:49 on dist-test-slave-j54z"
I20250109 20:59:49.378427 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:59:49.407162 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:59:49.408288 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:59:49.409724 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:59:49.411852 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:59:49.412050 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:49.412262 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:59:49.412411 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:59:49.450181 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:34079
I20250109 20:59:49.450284 21586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:34079 every 8 connection(s)
I20250109 20:59:49.462226 21587 heartbeater.cc:346] Connected to a master server at 127.17.14.190:36697
I20250109 20:59:49.462569 21587 heartbeater.cc:463] Registering TS with master...
I20250109 20:59:49.463235 21587 heartbeater.cc:510] Master 127.17.14.190:36697 requested a full tablet report, sending...
I20250109 20:59:49.464942 21311 ts_manager.cc:194] Registered new tserver with Master: f7deb6d8db534064a603046070039e3a (127.17.14.131:34079)
I20250109 20:59:49.465252 17466 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012083513s
I20250109 20:59:49.466708 21311 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50652
I20250109 20:59:50.274765 21438 heartbeater.cc:502] Master 127.17.14.190:36697 was elected leader, sending a full tablet report...
I20250109 20:59:50.367138 21513 heartbeater.cc:502] Master 127.17.14.190:36697 was elected leader, sending a full tablet report...
I20250109 20:59:50.469117 21587 heartbeater.cc:502] Master 127.17.14.190:36697 was elected leader, sending a full tablet report...
I20250109 20:59:50.498128 17466 test_util.cc:274] Using random seed: 1112725308
I20250109 20:59:50.518332 21311 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50654:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:59:50.520103 21311 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:59:50.561955 21403 tablet_service.cc:1467] Processing CreateTablet for tablet f510456fcc7845508b6e232c0fee9155 (DEFAULT_TABLE table=test-workload [id=5ebc2709711c4231a5cf4657932fe972]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:59:50.563297 21403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f510456fcc7845508b6e232c0fee9155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:50.567340 21478 tablet_service.cc:1467] Processing CreateTablet for tablet f510456fcc7845508b6e232c0fee9155 (DEFAULT_TABLE table=test-workload [id=5ebc2709711c4231a5cf4657932fe972]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:59:50.568282 21478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f510456fcc7845508b6e232c0fee9155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:50.570442 21552 tablet_service.cc:1467] Processing CreateTablet for tablet f510456fcc7845508b6e232c0fee9155 (DEFAULT_TABLE table=test-workload [id=5ebc2709711c4231a5cf4657932fe972]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:59:50.571552 21552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f510456fcc7845508b6e232c0fee9155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:59:50.581915 21607 tablet_bootstrap.cc:492] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d: Bootstrap starting.
I20250109 20:59:50.583181 21608 tablet_bootstrap.cc:492] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce: Bootstrap starting.
I20250109 20:59:50.587584 21609 tablet_bootstrap.cc:492] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a: Bootstrap starting.
I20250109 20:59:50.588022 21607 tablet_bootstrap.cc:654] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:50.589664 21608 tablet_bootstrap.cc:654] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:50.594359 21607 tablet_bootstrap.cc:492] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d: No bootstrap required, opened a new log
I20250109 20:59:50.594815 21608 tablet_bootstrap.cc:492] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce: No bootstrap required, opened a new log
I20250109 20:59:50.594866 21607 ts_tablet_manager.cc:1397] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20250109 20:59:50.595172 21609 tablet_bootstrap.cc:654] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a: Neither blocks nor log segments found. Creating new log.
I20250109 20:59:50.595441 21608 ts_tablet_manager.cc:1397] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20250109 20:59:50.598105 21608 raft_consensus.cc:357] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } }
I20250109 20:59:50.598003 21607 raft_consensus.cc:357] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } }
I20250109 20:59:50.598780 21608 raft_consensus.cc:383] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:50.598956 21607 raft_consensus.cc:383] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:50.599109 21608 raft_consensus.cc:738] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549513f3278c46feba3073887daecfce, State: Initialized, Role: FOLLOWER
I20250109 20:59:50.599407 21609 tablet_bootstrap.cc:492] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a: No bootstrap required, opened a new log
I20250109 20:59:50.599246 21607 raft_consensus.cc:738] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981558576c57415fae830c97bf70581d, State: Initialized, Role: FOLLOWER
I20250109 20:59:50.599851 21609 ts_tablet_manager.cc:1397] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20250109 20:59:50.599848 21608 consensus_queue.cc:260] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } }
I20250109 20:59:50.600136 21607 consensus_queue.cc:260] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } }
I20250109 20:59:50.603111 21609 raft_consensus.cc:357] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } }
I20250109 20:59:50.604104 21609 raft_consensus.cc:383] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:59:50.604418 21609 raft_consensus.cc:738] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7deb6d8db534064a603046070039e3a, State: Initialized, Role: FOLLOWER
I20250109 20:59:50.605136 21609 consensus_queue.cc:260] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } }
I20250109 20:59:50.606737 21607 ts_tablet_manager.cc:1428] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20250109 20:59:50.609088 21608 ts_tablet_manager.cc:1428] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20250109 20:59:50.612186 21609 ts_tablet_manager.cc:1428] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a: Time spent starting tablet: real 0.012s user 0.003s sys 0.007s
I20250109 20:59:50.682376 21614 raft_consensus.cc:491] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:59:50.682785 21614 raft_consensus.cc:513] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } }
I20250109 20:59:50.684774 21614 leader_election.cc:290] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 549513f3278c46feba3073887daecfce (127.17.14.130:42503), f7deb6d8db534064a603046070039e3a (127.17.14.131:34079)
I20250109 20:59:50.695179 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f510456fcc7845508b6e232c0fee9155" candidate_uuid: "981558576c57415fae830c97bf70581d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549513f3278c46feba3073887daecfce" is_pre_election: true
I20250109 20:59:50.695909 21488 raft_consensus.cc:2463] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 981558576c57415fae830c97bf70581d in term 0.
I20250109 20:59:50.695866 21562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f510456fcc7845508b6e232c0fee9155" candidate_uuid: "981558576c57415fae830c97bf70581d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7deb6d8db534064a603046070039e3a" is_pre_election: true
I20250109 20:59:50.696676 21562 raft_consensus.cc:2463] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 981558576c57415fae830c97bf70581d in term 0.
I20250109 20:59:50.697127 21378 leader_election.cc:304] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549513f3278c46feba3073887daecfce, 981558576c57415fae830c97bf70581d; no voters:
I20250109 20:59:50.697885 21614 raft_consensus.cc:2798] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:59:50.698207 21614 raft_consensus.cc:491] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:59:50.698441 21614 raft_consensus.cc:3054] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:50.702987 21614 raft_consensus.cc:513] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } }
I20250109 20:59:50.704392 21614 leader_election.cc:290] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [CANDIDATE]: Term 1 election: Requested vote from peers 549513f3278c46feba3073887daecfce (127.17.14.130:42503), f7deb6d8db534064a603046070039e3a (127.17.14.131:34079)
W20250109 20:59:50.704996 21588 tablet.cc:2367] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 20:59:50.705080 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f510456fcc7845508b6e232c0fee9155" candidate_uuid: "981558576c57415fae830c97bf70581d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549513f3278c46feba3073887daecfce"
I20250109 20:59:50.705379 21562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f510456fcc7845508b6e232c0fee9155" candidate_uuid: "981558576c57415fae830c97bf70581d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7deb6d8db534064a603046070039e3a"
I20250109 20:59:50.705724 21488 raft_consensus.cc:3054] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:50.705941 21562 raft_consensus.cc:3054] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:59:50.710245 21488 raft_consensus.cc:2463] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 981558576c57415fae830c97bf70581d in term 1.
I20250109 20:59:50.710397 21562 raft_consensus.cc:2463] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 981558576c57415fae830c97bf70581d in term 1.
I20250109 20:59:50.711071 21378 leader_election.cc:304] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549513f3278c46feba3073887daecfce, 981558576c57415fae830c97bf70581d; no voters:
I20250109 20:59:50.711807 21614 raft_consensus.cc:2798] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:59:50.713056 21614 raft_consensus.cc:695] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 1 LEADER]: Becoming Leader. State: Replica: 981558576c57415fae830c97bf70581d, State: Running, Role: LEADER
I20250109 20:59:50.713933 21614 consensus_queue.cc:237] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } }
I20250109 20:59:50.725919 21310 catalog_manager.cc:5526] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d reported cstate change: term changed from 0 to 1, leader changed from <none> to 981558576c57415fae830c97bf70581d (127.17.14.129). New cstate: current_term: 1 leader_uuid: "981558576c57415fae830c97bf70581d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20250109 20:59:50.751055 17466 tablet_server.cc:178] TabletServer@127.17.14.129:0 shutting down...
I20250109 20:59:50.768005 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 20:59:50.769830 17466 tablet_replica.cc:331] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d: stopping tablet replica
I20250109 20:59:50.770383 17466 raft_consensus.cc:2238] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:59:50.770928 17466 pending_rounds.cc:62] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d: Trying to abort 1 pending ops.
I20250109 20:59:50.771159 17466 pending_rounds.cc:69] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112525376384753664 op_type: NO_OP noop_request { }
I20250109 20:59:50.771490 17466 raft_consensus.cc:2883] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:59:50.771771 17466 raft_consensus.cc:2267] T f510456fcc7845508b6e232c0fee9155 P 981558576c57415fae830c97bf70581d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:59:50.790431 17466 tablet_server.cc:195] TabletServer@127.17.14.129:0 shutdown complete.
I20250109 20:59:51.039748 21364 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:51.040908 21364 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:51.041188 21364 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:52.588018 21620 raft_consensus.cc:491] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:59:52.588469 21620 raft_consensus.cc:513] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } }
I20250109 20:59:52.590283 21620 leader_election.cc:290] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 549513f3278c46feba3073887daecfce (127.17.14.130:42503), 981558576c57415fae830c97bf70581d (127.17.14.129:41401)
W20250109 20:59:52.596607 21525 leader_election.cc:336] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 981558576c57415fae830c97bf70581d (127.17.14.129:41401): Network error: Client connection negotiation failed: client connection to 127.17.14.129:41401: connect: Connection refused (error 111)
I20250109 20:59:52.599313 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f510456fcc7845508b6e232c0fee9155" candidate_uuid: "f7deb6d8db534064a603046070039e3a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549513f3278c46feba3073887daecfce" is_pre_election: true
I20250109 20:59:52.599848 21488 raft_consensus.cc:2463] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7deb6d8db534064a603046070039e3a in term 1.
I20250109 20:59:52.600719 21527 leader_election.cc:304] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 549513f3278c46feba3073887daecfce, f7deb6d8db534064a603046070039e3a; no voters: 981558576c57415fae830c97bf70581d
I20250109 20:59:52.601336 21620 raft_consensus.cc:2798] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250109 20:59:52.601642 21620 raft_consensus.cc:491] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:59:52.601907 21620 raft_consensus.cc:3054] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:59:52.606185 21620 raft_consensus.cc:513] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } }
I20250109 20:59:52.607553 21620 leader_election.cc:290] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [CANDIDATE]: Term 2 election: Requested vote from peers 549513f3278c46feba3073887daecfce (127.17.14.130:42503), 981558576c57415fae830c97bf70581d (127.17.14.129:41401)
I20250109 20:59:52.608193 21488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f510456fcc7845508b6e232c0fee9155" candidate_uuid: "f7deb6d8db534064a603046070039e3a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549513f3278c46feba3073887daecfce"
I20250109 20:59:52.608633 21488 raft_consensus.cc:3054] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [term 1 FOLLOWER]: Advancing to term 2
W20250109 20:59:52.610649 21525 leader_election.cc:336] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 981558576c57415fae830c97bf70581d (127.17.14.129:41401): Network error: Client connection negotiation failed: client connection to 127.17.14.129:41401: connect: Connection refused (error 111)
I20250109 20:59:52.612834 21488 raft_consensus.cc:2463] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7deb6d8db534064a603046070039e3a in term 2.
I20250109 20:59:52.613673 21527 leader_election.cc:304] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 549513f3278c46feba3073887daecfce, f7deb6d8db534064a603046070039e3a; no voters: 981558576c57415fae830c97bf70581d
I20250109 20:59:52.614275 21620 raft_consensus.cc:2798] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 2 FOLLOWER]: Leader election won for term 2
I20250109 20:59:52.615259 21620 raft_consensus.cc:695] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 2 LEADER]: Becoming Leader. State: Replica: f7deb6d8db534064a603046070039e3a, State: Running, Role: LEADER
I20250109 20:59:52.615984 21620 consensus_queue.cc:237] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } }
I20250109 20:59:52.621614 21310 catalog_manager.cc:5526] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a reported cstate change: term changed from 1 to 2, leader changed from 981558576c57415fae830c97bf70581d (127.17.14.129) to f7deb6d8db534064a603046070039e3a (127.17.14.131). New cstate: current_term: 2 leader_uuid: "f7deb6d8db534064a603046070039e3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "981558576c57415fae830c97bf70581d" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 41401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7deb6d8db534064a603046070039e3a" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 34079 } health_report { overall_health: HEALTHY } } }
I20250109 20:59:53.041977 21364 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:53.043501 21364 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:53.043807 21364 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 20:59:53.073575 21488 raft_consensus.cc:1270] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [term 2 FOLLOWER]: Refusing update from remote peer f7deb6d8db534064a603046070039e3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250109 20:59:53.074842 21620 consensus_queue.cc:1035] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "549513f3278c46feba3073887daecfce" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 20:59:53.088115 21525 consensus_peers.cc:487] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a -> Peer 981558576c57415fae830c97bf70581d (127.17.14.129:41401): Couldn't send request to peer 981558576c57415fae830c97bf70581d. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:41401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250109 20:59:53.993759 21525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.14.129:41401: connect: Connection refused (error 111) [suppressed 5 similar messages]
I20250109 20:59:55.044862 21364 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:55.046511 21364 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:55.046867 21364 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250109 20:59:55.625722 21525 consensus_peers.cc:487] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a -> Peer 981558576c57415fae830c97bf70581d (127.17.14.129:41401): Couldn't send request to peer 981558576c57415fae830c97bf70581d. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:41401: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250109 20:59:57.047708 21364 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:57.048885 21364 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:57.049113 21364 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250109 20:59:58.271114 21525 consensus_peers.cc:487] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a -> Peer 981558576c57415fae830c97bf70581d (127.17.14.129:41401): Couldn't send request to peer 981558576c57415fae830c97bf70581d. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:41401: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250109 20:59:59.049928 21364 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 20:59:59.051131 21364 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 20:59:59.051442 21364 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250109 20:59:59.223543 21525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.14.129:41401: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250109 21:00:00.812495 21525 consensus_peers.cc:487] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a -> Peer 981558576c57415fae830c97bf70581d (127.17.14.129:41401): Couldn't send request to peer 981558576c57415fae830c97bf70581d. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:41401: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250109 21:00:01.052361 21364 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250109 21:00:01.053597 21364 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250109 21:00:01.053920 21364 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250109 21:00:01.859899 17466 tablet_server.cc:178] TabletServer@127.17.14.130:0 shutting down...
I20250109 21:00:01.882035 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:01.882824 17466 tablet_replica.cc:331] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce: stopping tablet replica
I20250109 21:00:01.883451 17466 raft_consensus.cc:2238] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:01.884028 17466 raft_consensus.cc:2267] T f510456fcc7845508b6e232c0fee9155 P 549513f3278c46feba3073887daecfce [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:01.903611 17466 tablet_server.cc:195] TabletServer@127.17.14.130:0 shutdown complete.
I20250109 21:00:01.914345 17466 tablet_server.cc:178] TabletServer@127.17.14.131:0 shutting down...
I20250109 21:00:01.935271 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:01.935899 17466 tablet_replica.cc:331] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a: stopping tablet replica
I20250109 21:00:01.936497 17466 raft_consensus.cc:2238] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 2 LEADER]: Raft consensus shutting down.
I20250109 21:00:01.937295 17466 raft_consensus.cc:2267] T f510456fcc7845508b6e232c0fee9155 P f7deb6d8db534064a603046070039e3a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:01.956156 17466 tablet_server.cc:195] TabletServer@127.17.14.131:0 shutdown complete.
I20250109 21:00:01.967000 17466 master.cc:537] Master@127.17.14.190:36697 shutting down...
I20250109 21:00:01.984066 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:01.984742 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:01.985085 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P ede37d3c23d4484d95f5d93714530ec1: stopping tablet replica
I20250109 21:00:02.003412 17466 master.cc:559] Master@127.17.14.190:36697 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (13119 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20250109 21:00:02.038086 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:43287
I20250109 21:00:02.039201 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:02.045284 21656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:02.046276 21657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:02.047225 21659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:02.048194 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:02.048940 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:02.049134 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:02.049273 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456402049254 us; error 0 us; skew 500 ppm
I20250109 21:00:02.049804 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:02.052188 17466 webserver.cc:458] Webserver started at http://127.17.14.190:35869/ using document root <none> and password file <none>
I20250109 21:00:02.052645 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:02.052821 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:02.053066 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:02.054421 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "ff989a757ef74f3fa22bb4354b0fd85f"
format_stamp: "Formatted at 2025-01-09 21:00:02 on dist-test-slave-j54z"
I20250109 21:00:02.058873 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250109 21:00:02.062103 21664 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:02.062776 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:00:02.063030 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "ff989a757ef74f3fa22bb4354b0fd85f"
format_stamp: "Formatted at 2025-01-09 21:00:02 on dist-test-slave-j54z"
I20250109 21:00:02.063284 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:02.072445 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:02.073590 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:02.109669 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:43287
I20250109 21:00:02.109771 21715 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:43287 every 8 connection(s)
I20250109 21:00:02.113649 21716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:02.124907 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f: Bootstrap starting.
I20250109 21:00:02.129535 21716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:02.133561 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f: No bootstrap required, opened a new log
I20250109 21:00:02.135519 21716 raft_consensus.cc:357] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff989a757ef74f3fa22bb4354b0fd85f" member_type: VOTER }
I20250109 21:00:02.135978 21716 raft_consensus.cc:383] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:02.136194 21716 raft_consensus.cc:738] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff989a757ef74f3fa22bb4354b0fd85f, State: Initialized, Role: FOLLOWER
I20250109 21:00:02.136778 21716 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff989a757ef74f3fa22bb4354b0fd85f" member_type: VOTER }
I20250109 21:00:02.137220 21716 raft_consensus.cc:397] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:00:02.137424 21716 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:00:02.137686 21716 raft_consensus.cc:3054] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:02.141930 21716 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff989a757ef74f3fa22bb4354b0fd85f" member_type: VOTER }
I20250109 21:00:02.142403 21716 leader_election.cc:304] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff989a757ef74f3fa22bb4354b0fd85f; no voters:
I20250109 21:00:02.143428 21716 leader_election.cc:290] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:00:02.143826 21719 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:02.145016 21719 raft_consensus.cc:695] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [term 1 LEADER]: Becoming Leader. State: Replica: ff989a757ef74f3fa22bb4354b0fd85f, State: Running, Role: LEADER
I20250109 21:00:02.145713 21719 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff989a757ef74f3fa22bb4354b0fd85f" member_type: VOTER }
I20250109 21:00:02.146347 21716 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:00:02.148514 21720 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff989a757ef74f3fa22bb4354b0fd85f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff989a757ef74f3fa22bb4354b0fd85f" member_type: VOTER } }
I20250109 21:00:02.149253 21720 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [sys.catalog]: This master's current role is: LEADER
I20250109 21:00:02.148511 21721 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff989a757ef74f3fa22bb4354b0fd85f. Latest consensus state: current_term: 1 leader_uuid: "ff989a757ef74f3fa22bb4354b0fd85f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff989a757ef74f3fa22bb4354b0fd85f" member_type: VOTER } }
I20250109 21:00:02.149629 21721 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [sys.catalog]: This master's current role is: LEADER
I20250109 21:00:02.154935 21726 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:00:02.161855 21726 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:00:02.162797 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 21:00:02.170217 21726 catalog_manager.cc:1348] Generated new cluster ID: 13ae5506ad0349bdb40d0a476c26a8cf
I20250109 21:00:02.170439 21726 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:00:02.201982 21726 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:00:02.203233 21726 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:00:02.230962 21726 catalog_manager.cc:5899] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f: Generated new TSK 0
I20250109 21:00:02.231479 21726 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:00:02.294391 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:02.300925 21737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:02.301923 21738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:02.302577 21740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:02.303817 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:02.304522 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:02.304723 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:02.304847 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456402304836 us; error 0 us; skew 500 ppm
I20250109 21:00:02.305264 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:02.310333 17466 webserver.cc:458] Webserver started at http://127.17.14.129:41071/ using document root <none> and password file <none>
I20250109 21:00:02.310834 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:02.310986 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:02.311178 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:02.312238 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-0-root/instance:
uuid: "cf15125b30b34f7291283d2ebc3ef969"
format_stamp: "Formatted at 2025-01-09 21:00:02 on dist-test-slave-j54z"
I20250109 21:00:02.316628 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250109 21:00:02.319713 21745 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:02.320417 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250109 21:00:02.320686 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "cf15125b30b34f7291283d2ebc3ef969"
format_stamp: "Formatted at 2025-01-09 21:00:02 on dist-test-slave-j54z"
I20250109 21:00:02.320950 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:02.345592 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:02.346695 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:02.348196 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:02.350477 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:00:02.350683 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:02.350888 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:00:02.351034 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:02.389621 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:34149
I20250109 21:00:02.389715 21807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:34149 every 8 connection(s)
I20250109 21:00:02.393965 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:02.402274 21812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:02.402670 21813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:02.403949 21815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:02.404944 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:02.406126 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:02.406385 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:02.406565 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456402406546 us; error 0 us; skew 500 ppm
I20250109 21:00:02.407196 21808 heartbeater.cc:346] Connected to a master server at 127.17.14.190:43287
I20250109 21:00:02.407229 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:02.407657 21808 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:02.408334 21808 heartbeater.cc:510] Master 127.17.14.190:43287 requested a full tablet report, sending...
I20250109 21:00:02.410102 17466 webserver.cc:458] Webserver started at http://127.17.14.130:44933/ using document root <none> and password file <none>
I20250109 21:00:02.410539 21681 ts_manager.cc:194] Registered new tserver with Master: cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149)
I20250109 21:00:02.410740 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:02.411022 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:02.411324 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:02.412859 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-1-root/instance:
uuid: "15e7316da760420083b0ee12e34ab3f8"
format_stamp: "Formatted at 2025-01-09 21:00:02 on dist-test-slave-j54z"
I20250109 21:00:02.412962 21681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48130
I20250109 21:00:02.417356 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250109 21:00:02.420538 21820 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:02.421221 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250109 21:00:02.421460 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "15e7316da760420083b0ee12e34ab3f8"
format_stamp: "Formatted at 2025-01-09 21:00:02 on dist-test-slave-j54z"
I20250109 21:00:02.421756 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:02.441917 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:02.442989 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:02.444324 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:02.446456 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:00:02.446650 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:02.446856 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:00:02.446997 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:02.484676 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:38587
I20250109 21:00:02.484730 21882 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:38587 every 8 connection(s)
I20250109 21:00:02.489439 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:02.497046 21886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:02.497859 21887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:02.499327 21889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:02.500646 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:02.500874 21883 heartbeater.cc:346] Connected to a master server at 127.17.14.190:43287
I20250109 21:00:02.501258 21883 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:02.501560 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:02.501765 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:02.501935 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456402501915 us; error 0 us; skew 500 ppm
I20250109 21:00:02.501998 21883 heartbeater.cc:510] Master 127.17.14.190:43287 requested a full tablet report, sending...
I20250109 21:00:02.502447 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:02.504053 21681 ts_manager.cc:194] Registered new tserver with Master: 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587)
I20250109 21:00:02.505095 17466 webserver.cc:458] Webserver started at http://127.17.14.131:41203/ using document root <none> and password file <none>
I20250109 21:00:02.505764 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:02.505982 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:02.505930 21681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48146
I20250109 21:00:02.506402 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:02.507647 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-2-root/instance:
uuid: "77b28859ea234bdfa21ca7e9fe9517ef"
format_stamp: "Formatted at 2025-01-09 21:00:02 on dist-test-slave-j54z"
I20250109 21:00:02.511878 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 21:00:02.514844 21894 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:02.515630 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:00:02.515856 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "77b28859ea234bdfa21ca7e9fe9517ef"
format_stamp: "Formatted at 2025-01-09 21:00:02 on dist-test-slave-j54z"
I20250109 21:00:02.516100 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:02.532022 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:02.533062 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:02.534404 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:02.536473 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:00:02.536724 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:02.537007 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:00:02.537204 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:02.574709 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:39529
I20250109 21:00:02.574790 21956 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:39529 every 8 connection(s)
I20250109 21:00:02.586853 21957 heartbeater.cc:346] Connected to a master server at 127.17.14.190:43287
I20250109 21:00:02.587133 21957 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:02.587745 21957 heartbeater.cc:510] Master 127.17.14.190:43287 requested a full tablet report, sending...
I20250109 21:00:02.589421 21681 ts_manager.cc:194] Registered new tserver with Master: 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131:39529)
I20250109 21:00:02.589706 17466 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012024768s
I20250109 21:00:02.590721 21681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48158
I20250109 21:00:03.415550 21808 heartbeater.cc:502] Master 127.17.14.190:43287 was elected leader, sending a full tablet report...
I20250109 21:00:03.508420 21883 heartbeater.cc:502] Master 127.17.14.190:43287 was elected leader, sending a full tablet report...
I20250109 21:00:03.592908 21957 heartbeater.cc:502] Master 127.17.14.190:43287 was elected leader, sending a full tablet report...
I20250109 21:00:03.621745 17466 test_util.cc:274] Using random seed: 1125848923
I20250109 21:00:03.643103 21681 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48162:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:00:03.645371 21681 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:00:03.697069 21848 tablet_service.cc:1467] Processing CreateTablet for tablet c9b3f352d7ae4b1d9a144efaad03a658 (DEFAULT_TABLE table=test-workload [id=41198418dc344bd981bbbbbc6a95ad51]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 21:00:03.698176 21922 tablet_service.cc:1467] Processing CreateTablet for tablet c9b3f352d7ae4b1d9a144efaad03a658 (DEFAULT_TABLE table=test-workload [id=41198418dc344bd981bbbbbc6a95ad51]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 21:00:03.698575 21848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b3f352d7ae4b1d9a144efaad03a658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:03.699460 21922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b3f352d7ae4b1d9a144efaad03a658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:03.701959 21921 tablet_service.cc:1467] Processing CreateTablet for tablet 7ebe136f04cb4e6eae1fe43a99f49c0e (DEFAULT_TABLE table=test-workload [id=41198418dc344bd981bbbbbc6a95ad51]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250109 21:00:03.703240 21921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ebe136f04cb4e6eae1fe43a99f49c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:03.697069 21847 tablet_service.cc:1467] Processing CreateTablet for tablet 7ebe136f04cb4e6eae1fe43a99f49c0e (DEFAULT_TABLE table=test-workload [id=41198418dc344bd981bbbbbc6a95ad51]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250109 21:00:03.706848 21847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ebe136f04cb4e6eae1fe43a99f49c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:03.710570 21773 tablet_service.cc:1467] Processing CreateTablet for tablet c9b3f352d7ae4b1d9a144efaad03a658 (DEFAULT_TABLE table=test-workload [id=41198418dc344bd981bbbbbc6a95ad51]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 21:00:03.711833 21773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b3f352d7ae4b1d9a144efaad03a658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:03.719311 21772 tablet_service.cc:1467] Processing CreateTablet for tablet 7ebe136f04cb4e6eae1fe43a99f49c0e (DEFAULT_TABLE table=test-workload [id=41198418dc344bd981bbbbbc6a95ad51]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250109 21:00:03.720933 21772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ebe136f04cb4e6eae1fe43a99f49c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:03.721833 21920 tablet_service.cc:1467] Processing CreateTablet for tablet 7287b207fde4413aae50e6346c490826 (DEFAULT_TABLE table=test-workload [id=41198418dc344bd981bbbbbc6a95ad51]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 21:00:03.723101 21920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7287b207fde4413aae50e6346c490826. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:03.724996 21846 tablet_service.cc:1467] Processing CreateTablet for tablet 7287b207fde4413aae50e6346c490826 (DEFAULT_TABLE table=test-workload [id=41198418dc344bd981bbbbbc6a95ad51]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 21:00:03.726316 21846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7287b207fde4413aae50e6346c490826. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:03.726122 21845 tablet_service.cc:1467] Processing CreateTablet for tablet 721d29157db9464e96715e12031235e5 (DEFAULT_TABLE table=test-workload [id=41198418dc344bd981bbbbbc6a95ad51]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 21:00:03.727329 21845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 721d29157db9464e96715e12031235e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:03.728431 21919 tablet_service.cc:1467] Processing CreateTablet for tablet 721d29157db9464e96715e12031235e5 (DEFAULT_TABLE table=test-workload [id=41198418dc344bd981bbbbbc6a95ad51]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 21:00:03.729722 21919 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 721d29157db9464e96715e12031235e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:03.743233 21771 tablet_service.cc:1467] Processing CreateTablet for tablet 7287b207fde4413aae50e6346c490826 (DEFAULT_TABLE table=test-workload [id=41198418dc344bd981bbbbbc6a95ad51]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 21:00:03.744611 21771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7287b207fde4413aae50e6346c490826. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:03.747677 21977 tablet_bootstrap.cc:492] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef: Bootstrap starting.
I20250109 21:00:03.751073 21770 tablet_service.cc:1467] Processing CreateTablet for tablet 721d29157db9464e96715e12031235e5 (DEFAULT_TABLE table=test-workload [id=41198418dc344bd981bbbbbc6a95ad51]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 21:00:03.752314 21770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 721d29157db9464e96715e12031235e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:03.756191 21977 tablet_bootstrap.cc:654] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:03.778712 21978 tablet_bootstrap.cc:492] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8: Bootstrap starting.
I20250109 21:00:03.786077 21977 tablet_bootstrap.cc:492] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef: No bootstrap required, opened a new log
I20250109 21:00:03.786579 21977 ts_tablet_manager.cc:1397] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.013s
I20250109 21:00:03.788328 21980 tablet_bootstrap.cc:492] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969: Bootstrap starting.
I20250109 21:00:03.789364 21977 raft_consensus.cc:357] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } }
I20250109 21:00:03.790213 21977 raft_consensus.cc:383] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:03.790498 21977 raft_consensus.cc:738] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b28859ea234bdfa21ca7e9fe9517ef, State: Initialized, Role: FOLLOWER
I20250109 21:00:03.791210 21977 consensus_queue.cc:260] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } }
I20250109 21:00:03.795361 21977 ts_tablet_manager.cc:1428] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250109 21:00:03.796135 21980 tablet_bootstrap.cc:654] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:03.796352 21977 tablet_bootstrap.cc:492] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef: Bootstrap starting.
I20250109 21:00:03.797878 21978 tablet_bootstrap.cc:654] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:03.804343 21977 tablet_bootstrap.cc:654] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:03.810281 21980 tablet_bootstrap.cc:492] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969: No bootstrap required, opened a new log
I20250109 21:00:03.810412 21978 tablet_bootstrap.cc:492] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8: No bootstrap required, opened a new log
I20250109 21:00:03.810514 21977 tablet_bootstrap.cc:492] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef: No bootstrap required, opened a new log
I20250109 21:00:03.810771 21980 ts_tablet_manager.cc:1397] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.008s
I20250109 21:00:03.810889 21978 ts_tablet_manager.cc:1397] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.014s
I20250109 21:00:03.810989 21977 ts_tablet_manager.cc:1397] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20250109 21:00:03.813724 21978 raft_consensus.cc:357] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.813972 21980 raft_consensus.cc:357] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } }
I20250109 21:00:03.814527 21978 raft_consensus.cc:383] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:03.814255 21977 raft_consensus.cc:357] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.814687 21980 raft_consensus.cc:383] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:03.814896 21978 raft_consensus.cc:738] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e7316da760420083b0ee12e34ab3f8, State: Initialized, Role: FOLLOWER
I20250109 21:00:03.815050 21977 raft_consensus.cc:383] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:03.815104 21980 raft_consensus.cc:738] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf15125b30b34f7291283d2ebc3ef969, State: Initialized, Role: FOLLOWER
I20250109 21:00:03.815490 21977 raft_consensus.cc:738] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b28859ea234bdfa21ca7e9fe9517ef, State: Initialized, Role: FOLLOWER
I20250109 21:00:03.815778 21978 consensus_queue.cc:260] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.815974 21980 consensus_queue.cc:260] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } }
I20250109 21:00:03.816176 21977 consensus_queue.cc:260] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.818374 21977 ts_tablet_manager.cc:1428] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250109 21:00:03.819319 21977 tablet_bootstrap.cc:492] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef: Bootstrap starting.
I20250109 21:00:03.825770 21980 ts_tablet_manager.cc:1428] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20250109 21:00:03.826005 21977 tablet_bootstrap.cc:654] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:03.825773 21978 ts_tablet_manager.cc:1428] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20250109 21:00:03.829178 21978 tablet_bootstrap.cc:492] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8: Bootstrap starting.
I20250109 21:00:03.835330 21978 tablet_bootstrap.cc:654] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:03.836036 21977 tablet_bootstrap.cc:492] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef: No bootstrap required, opened a new log
I20250109 21:00:03.836551 21977 ts_tablet_manager.cc:1397] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.002s
I20250109 21:00:03.839177 21977 raft_consensus.cc:357] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.839779 21981 raft_consensus.cc:491] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:03.840023 21977 raft_consensus.cc:383] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:03.840466 21977 raft_consensus.cc:738] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b28859ea234bdfa21ca7e9fe9517ef, State: Initialized, Role: FOLLOWER
I20250109 21:00:03.840240 21981 raft_consensus.cc:513] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } }
I20250109 21:00:03.841434 21977 consensus_queue.cc:260] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.843678 21981 leader_election.cc:290] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587), cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149)
I20250109 21:00:03.845355 21980 tablet_bootstrap.cc:492] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969: Bootstrap starting.
I20250109 21:00:03.862170 21978 tablet_bootstrap.cc:492] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8: No bootstrap required, opened a new log
I20250109 21:00:03.862825 21978 ts_tablet_manager.cc:1397] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.009s
I20250109 21:00:03.866007 21978 raft_consensus.cc:357] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } }
I20250109 21:00:03.866904 21978 raft_consensus.cc:383] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:03.867271 21978 raft_consensus.cc:738] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e7316da760420083b0ee12e34ab3f8, State: Initialized, Role: FOLLOWER
I20250109 21:00:03.868161 21978 consensus_queue.cc:260] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } }
I20250109 21:00:03.871439 21980 tablet_bootstrap.cc:654] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:03.872973 21977 ts_tablet_manager.cc:1428] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef: Time spent starting tablet: real 0.036s user 0.009s sys 0.007s
I20250109 21:00:03.873958 21978 ts_tablet_manager.cc:1428] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250109 21:00:03.874120 21977 tablet_bootstrap.cc:492] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef: Bootstrap starting.
I20250109 21:00:03.880252 21978 tablet_bootstrap.cc:492] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8: Bootstrap starting.
I20250109 21:00:03.882059 21977 tablet_bootstrap.cc:654] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:03.885102 21858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b3f352d7ae4b1d9a144efaad03a658" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e7316da760420083b0ee12e34ab3f8" is_pre_election: true
I20250109 21:00:03.886003 21858 raft_consensus.cc:2463] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b28859ea234bdfa21ca7e9fe9517ef in term 0.
I20250109 21:00:03.886983 21978 tablet_bootstrap.cc:654] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:03.887476 21898 leader_election.cc:304] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e7316da760420083b0ee12e34ab3f8, 77b28859ea234bdfa21ca7e9fe9517ef; no voters:
I20250109 21:00:03.888605 21991 raft_consensus.cc:2798] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:03.888993 21991 raft_consensus.cc:491] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:03.889070 21981 raft_consensus.cc:491] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:03.889384 21991 raft_consensus.cc:3054] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:03.889566 21981 raft_consensus.cc:513] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.891212 21980 tablet_bootstrap.cc:492] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969: No bootstrap required, opened a new log
I20250109 21:00:03.891651 21981 leader_election.cc:290] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149), 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587)
I20250109 21:00:03.893249 21783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b3f352d7ae4b1d9a144efaad03a658" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf15125b30b34f7291283d2ebc3ef969" is_pre_election: true
I20250109 21:00:03.894183 21783 raft_consensus.cc:2463] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b28859ea234bdfa21ca7e9fe9517ef in term 0.
I20250109 21:00:03.895769 21782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebe136f04cb4e6eae1fe43a99f49c0e" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf15125b30b34f7291283d2ebc3ef969" is_pre_election: true
W20250109 21:00:03.898561 21897 leader_election.cc:343] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Illegal state: must be running to vote when last-logged opid is not known
I20250109 21:00:03.898902 21991 raft_consensus.cc:513] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } }
I20250109 21:00:03.899703 21858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebe136f04cb4e6eae1fe43a99f49c0e" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e7316da760420083b0ee12e34ab3f8" is_pre_election: true
I20250109 21:00:03.900569 21858 raft_consensus.cc:2463] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b28859ea234bdfa21ca7e9fe9517ef in term 0.
I20250109 21:00:03.902040 21980 ts_tablet_manager.cc:1397] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.009s
I20250109 21:00:03.902243 21898 leader_election.cc:304] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15e7316da760420083b0ee12e34ab3f8, 77b28859ea234bdfa21ca7e9fe9517ef; no voters: cf15125b30b34f7291283d2ebc3ef969
I20250109 21:00:03.903816 21981 raft_consensus.cc:2798] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:03.904279 21981 raft_consensus.cc:491] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:03.904675 21981 raft_consensus.cc:3054] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:03.905834 21980 raft_consensus.cc:357] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.906828 21980 raft_consensus.cc:383] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:03.907186 21980 raft_consensus.cc:738] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf15125b30b34f7291283d2ebc3ef969, State: Initialized, Role: FOLLOWER
I20250109 21:00:03.908000 21858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b3f352d7ae4b1d9a144efaad03a658" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e7316da760420083b0ee12e34ab3f8"
I20250109 21:00:03.908782 21858 raft_consensus.cc:3054] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:03.908174 21980 consensus_queue.cc:260] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.911154 21980 ts_tablet_manager.cc:1428] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969: Time spent starting tablet: real 0.009s user 0.001s sys 0.007s
I20250109 21:00:03.912065 21980 tablet_bootstrap.cc:492] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969: Bootstrap starting.
I20250109 21:00:03.919185 21980 tablet_bootstrap.cc:654] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:03.924974 21977 tablet_bootstrap.cc:492] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef: No bootstrap required, opened a new log
I20250109 21:00:03.925084 21782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b3f352d7ae4b1d9a144efaad03a658" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf15125b30b34f7291283d2ebc3ef969"
I20250109 21:00:03.925837 21977 ts_tablet_manager.cc:1397] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef: Time spent bootstrapping tablet: real 0.052s user 0.031s sys 0.003s
I20250109 21:00:03.926062 21782 raft_consensus.cc:3054] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:03.929458 21977 raft_consensus.cc:357] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.930377 21977 raft_consensus.cc:383] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:03.930735 21977 raft_consensus.cc:738] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b28859ea234bdfa21ca7e9fe9517ef, State: Initialized, Role: FOLLOWER
I20250109 21:00:03.931466 21980 tablet_bootstrap.cc:492] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969: No bootstrap required, opened a new log
I20250109 21:00:03.932008 21980 ts_tablet_manager.cc:1397] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.008s
I20250109 21:00:03.934463 21977 consensus_queue.cc:260] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.934770 21980 raft_consensus.cc:357] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.935686 21980 raft_consensus.cc:383] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:03.936102 21980 raft_consensus.cc:738] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf15125b30b34f7291283d2ebc3ef969, State: Initialized, Role: FOLLOWER
I20250109 21:00:03.936364 21782 raft_consensus.cc:2463] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b28859ea234bdfa21ca7e9fe9517ef in term 1.
I20250109 21:00:03.937268 21980 consensus_queue.cc:260] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.943977 21897 leader_election.cc:304] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77b28859ea234bdfa21ca7e9fe9517ef, cf15125b30b34f7291283d2ebc3ef969; no voters:
I20250109 21:00:03.944600 21980 ts_tablet_manager.cc:1428] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969: Time spent starting tablet: real 0.012s user 0.004s sys 0.002s
I20250109 21:00:03.945122 21996 raft_consensus.cc:2798] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:03.939013 21981 raft_consensus.cc:513] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.940225 21977 ts_tablet_manager.cc:1428] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef: Time spent starting tablet: real 0.014s user 0.002s sys 0.008s
I20250109 21:00:03.942014 21858 raft_consensus.cc:2463] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b28859ea234bdfa21ca7e9fe9517ef in term 1.
I20250109 21:00:03.942548 21991 leader_election.cc:290] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 election: Requested vote from peers 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587), cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149)
I20250109 21:00:03.948956 21782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebe136f04cb4e6eae1fe43a99f49c0e" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf15125b30b34f7291283d2ebc3ef969"
I20250109 21:00:03.949328 21978 tablet_bootstrap.cc:492] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8: No bootstrap required, opened a new log
I20250109 21:00:03.949199 21858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebe136f04cb4e6eae1fe43a99f49c0e" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e7316da760420083b0ee12e34ab3f8"
I20250109 21:00:03.949882 21978 ts_tablet_manager.cc:1397] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8: Time spent bootstrapping tablet: real 0.070s user 0.028s sys 0.034s
W20250109 21:00:03.950276 21897 leader_election.cc:343] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Illegal state: must be running to vote when last-logged opid is not known
I20250109 21:00:03.950556 21980 tablet_bootstrap.cc:492] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969: Bootstrap starting.
I20250109 21:00:03.949872 21858 raft_consensus.cc:3054] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:03.952816 21978 raft_consensus.cc:357] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.953639 21978 raft_consensus.cc:383] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:03.953966 21978 raft_consensus.cc:738] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e7316da760420083b0ee12e34ab3f8, State: Initialized, Role: FOLLOWER
I20250109 21:00:03.955087 21981 leader_election.cc:290] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 election: Requested vote from peers cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149), 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587)
I20250109 21:00:03.954728 21978 consensus_queue.cc:260] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.957386 21980 tablet_bootstrap.cc:654] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:03.957759 21978 ts_tablet_manager.cc:1428] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250109 21:00:03.958778 21978 tablet_bootstrap.cc:492] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8: Bootstrap starting.
I20250109 21:00:03.959123 21858 raft_consensus.cc:2463] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b28859ea234bdfa21ca7e9fe9517ef in term 1.
I20250109 21:00:03.960215 21898 leader_election.cc:304] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15e7316da760420083b0ee12e34ab3f8, 77b28859ea234bdfa21ca7e9fe9517ef; no voters: cf15125b30b34f7291283d2ebc3ef969
I20250109 21:00:03.961722 21981 raft_consensus.cc:2798] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:03.962450 21981 raft_consensus.cc:695] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Becoming Leader. State: Replica: 77b28859ea234bdfa21ca7e9fe9517ef, State: Running, Role: LEADER
I20250109 21:00:03.963454 21981 consensus_queue.cc:237] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.969345 21996 raft_consensus.cc:695] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Becoming Leader. State: Replica: 77b28859ea234bdfa21ca7e9fe9517ef, State: Running, Role: LEADER
I20250109 21:00:03.970618 21996 consensus_queue.cc:237] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } }
I20250109 21:00:03.971685 21978 tablet_bootstrap.cc:654] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:03.975543 21980 tablet_bootstrap.cc:492] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969: No bootstrap required, opened a new log
I20250109 21:00:03.976244 21980 ts_tablet_manager.cc:1397] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.007s
I20250109 21:00:03.979573 21980 raft_consensus.cc:357] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.980444 21980 raft_consensus.cc:383] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:03.980826 21980 raft_consensus.cc:738] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf15125b30b34f7291283d2ebc3ef969, State: Initialized, Role: FOLLOWER
I20250109 21:00:03.981629 21978 tablet_bootstrap.cc:492] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8: No bootstrap required, opened a new log
I20250109 21:00:03.982082 21978 ts_tablet_manager.cc:1397] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.003s
I20250109 21:00:03.981699 21980 consensus_queue.cc:260] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.984298 21980 ts_tablet_manager.cc:1428] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250109 21:00:03.984707 21978 raft_consensus.cc:357] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.985435 21978 raft_consensus.cc:383] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:03.985755 21978 raft_consensus.cc:738] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e7316da760420083b0ee12e34ab3f8, State: Initialized, Role: FOLLOWER
I20250109 21:00:03.986454 21978 consensus_queue.cc:260] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:03.988847 21978 ts_tablet_manager.cc:1428] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20250109 21:00:03.995837 21679 catalog_manager.cc:5526] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131). New cstate: current_term: 1 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:04.007850 21678 catalog_manager.cc:5526] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131). New cstate: current_term: 1 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: HEALTHY } } }
I20250109 21:00:04.014103 21986 raft_consensus.cc:491] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:04.014546 21986 raft_consensus.cc:513] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:04.016641 21986 leader_election.cc:290] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587), 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131:39529)
I20250109 21:00:04.020581 21986 raft_consensus.cc:491] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:04.021014 21986 raft_consensus.cc:513] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:04.022810 21986 leader_election.cc:290] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587), 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131:39529)
I20250109 21:00:04.027511 21858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebe136f04cb4e6eae1fe43a99f49c0e" candidate_uuid: "cf15125b30b34f7291283d2ebc3ef969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e7316da760420083b0ee12e34ab3f8" is_pre_election: true
I20250109 21:00:04.027554 21932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebe136f04cb4e6eae1fe43a99f49c0e" candidate_uuid: "cf15125b30b34f7291283d2ebc3ef969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" is_pre_election: true
I20250109 21:00:04.028403 21858 raft_consensus.cc:2388] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf15125b30b34f7291283d2ebc3ef969 in current term 1: Already voted for candidate 77b28859ea234bdfa21ca7e9fe9517ef in this term.
I20250109 21:00:04.028045 21931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287b207fde4413aae50e6346c490826" candidate_uuid: "cf15125b30b34f7291283d2ebc3ef969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" is_pre_election: true
I20250109 21:00:04.028072 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287b207fde4413aae50e6346c490826" candidate_uuid: "cf15125b30b34f7291283d2ebc3ef969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e7316da760420083b0ee12e34ab3f8" is_pre_election: true
I20250109 21:00:04.028867 21931 raft_consensus.cc:2463] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf15125b30b34f7291283d2ebc3ef969 in term 0.
I20250109 21:00:04.028936 21857 raft_consensus.cc:2463] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf15125b30b34f7291283d2ebc3ef969 in term 0.
I20250109 21:00:04.029820 21749 leader_election.cc:304] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf15125b30b34f7291283d2ebc3ef969; no voters: 15e7316da760420083b0ee12e34ab3f8, 77b28859ea234bdfa21ca7e9fe9517ef
I20250109 21:00:04.029970 21746 leader_election.cc:304] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77b28859ea234bdfa21ca7e9fe9517ef, cf15125b30b34f7291283d2ebc3ef969; no voters:
I20250109 21:00:04.030500 21986 raft_consensus.cc:3054] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:04.031626 22004 raft_consensus.cc:2798] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:04.031934 22004 raft_consensus.cc:491] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:04.032186 22004 raft_consensus.cc:3054] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:04.035696 21986 raft_consensus.cc:2743] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 21:00:04.036715 22004 raft_consensus.cc:513] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:04.038278 22004 leader_election.cc:290] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [CANDIDATE]: Term 1 election: Requested vote from peers 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587), 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131:39529)
I20250109 21:00:04.038985 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287b207fde4413aae50e6346c490826" candidate_uuid: "cf15125b30b34f7291283d2ebc3ef969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e7316da760420083b0ee12e34ab3f8"
I20250109 21:00:04.039489 21857 raft_consensus.cc:3054] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:04.039448 21931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287b207fde4413aae50e6346c490826" candidate_uuid: "cf15125b30b34f7291283d2ebc3ef969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b28859ea234bdfa21ca7e9fe9517ef"
I20250109 21:00:04.039971 21931 raft_consensus.cc:3054] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:04.043991 21857 raft_consensus.cc:2463] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf15125b30b34f7291283d2ebc3ef969 in term 1.
I20250109 21:00:04.044237 21931 raft_consensus.cc:2463] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf15125b30b34f7291283d2ebc3ef969 in term 1.
I20250109 21:00:04.044879 21749 leader_election.cc:304] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e7316da760420083b0ee12e34ab3f8, cf15125b30b34f7291283d2ebc3ef969; no voters:
I20250109 21:00:04.045507 22004 raft_consensus.cc:2798] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:04.045877 22004 raft_consensus.cc:695] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 1 LEADER]: Becoming Leader. State: Replica: cf15125b30b34f7291283d2ebc3ef969, State: Running, Role: LEADER
I20250109 21:00:04.046571 22004 consensus_queue.cc:237] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:04.052827 21678 catalog_manager.cc:5526] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129). New cstate: current_term: 1 leader_uuid: "cf15125b30b34f7291283d2ebc3ef969" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:04.126356 21996 raft_consensus.cc:491] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:04.127189 21996 raft_consensus.cc:513] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:04.129379 21996 leader_election.cc:290] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149), 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587)
I20250109 21:00:04.131078 21782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721d29157db9464e96715e12031235e5" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf15125b30b34f7291283d2ebc3ef969" is_pre_election: true
I20250109 21:00:04.131191 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721d29157db9464e96715e12031235e5" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e7316da760420083b0ee12e34ab3f8" is_pre_election: true
I20250109 21:00:04.132140 21782 raft_consensus.cc:2463] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b28859ea234bdfa21ca7e9fe9517ef in term 0.
I20250109 21:00:04.132206 21857 raft_consensus.cc:2463] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b28859ea234bdfa21ca7e9fe9517ef in term 0.
I20250109 21:00:04.133364 21897 leader_election.cc:304] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77b28859ea234bdfa21ca7e9fe9517ef, cf15125b30b34f7291283d2ebc3ef969; no voters:
I20250109 21:00:04.134083 21996 raft_consensus.cc:2798] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:04.134435 21996 raft_consensus.cc:491] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:04.134676 21996 raft_consensus.cc:3054] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:04.141273 21996 raft_consensus.cc:513] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:04.142861 21996 leader_election.cc:290] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 election: Requested vote from peers cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149), 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587)
I20250109 21:00:04.143606 21782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721d29157db9464e96715e12031235e5" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf15125b30b34f7291283d2ebc3ef969"
I20250109 21:00:04.143867 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721d29157db9464e96715e12031235e5" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e7316da760420083b0ee12e34ab3f8"
I20250109 21:00:04.144114 21782 raft_consensus.cc:3054] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:04.144369 21857 raft_consensus.cc:3054] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:04.150679 21782 raft_consensus.cc:2463] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b28859ea234bdfa21ca7e9fe9517ef in term 1.
I20250109 21:00:04.151765 21897 leader_election.cc:304] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77b28859ea234bdfa21ca7e9fe9517ef, cf15125b30b34f7291283d2ebc3ef969; no voters:
I20250109 21:00:04.152618 21996 raft_consensus.cc:2798] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:04.153105 21996 raft_consensus.cc:695] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Becoming Leader. State: Replica: 77b28859ea234bdfa21ca7e9fe9517ef, State: Running, Role: LEADER
I20250109 21:00:04.153353 21857 raft_consensus.cc:2463] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b28859ea234bdfa21ca7e9fe9517ef in term 1.
I20250109 21:00:04.154603 21996 consensus_queue.cc:237] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
W20250109 21:00:04.161362 21729 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250109 21:00:04.163118 21678 catalog_manager.cc:5526] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131). New cstate: current_term: 1 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: HEALTHY } } }
I20250109 21:00:04.266763 17466 tablet_server.cc:178] TabletServer@127.17.14.129:0 shutting down...
I20250109 21:00:04.288692 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:04.289808 17466 tablet_replica.cc:331] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969: stopping tablet replica
I20250109 21:00:04.290534 17466 raft_consensus.cc:2238] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:04.290987 17466 raft_consensus.cc:2267] T 721d29157db9464e96715e12031235e5 P cf15125b30b34f7291283d2ebc3ef969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:04.293509 17466 tablet_replica.cc:331] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969: stopping tablet replica
I20250109 21:00:04.294019 17466 raft_consensus.cc:2238] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:04.294807 17466 pending_rounds.cc:62] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969: Trying to abort 1 pending ops.
I20250109 21:00:04.295007 17466 pending_rounds.cc:69] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112525430981910528 op_type: NO_OP noop_request { }
I20250109 21:00:04.295359 17466 raft_consensus.cc:2883] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 21:00:04.295647 17466 raft_consensus.cc:2267] T 7287b207fde4413aae50e6346c490826 P cf15125b30b34f7291283d2ebc3ef969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:04.298415 17466 tablet_replica.cc:331] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969: stopping tablet replica
I20250109 21:00:04.298945 17466 raft_consensus.cc:2238] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:04.299396 17466 raft_consensus.cc:2267] T 7ebe136f04cb4e6eae1fe43a99f49c0e P cf15125b30b34f7291283d2ebc3ef969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:04.301625 17466 tablet_replica.cc:331] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969: stopping tablet replica
I20250109 21:00:04.302090 17466 raft_consensus.cc:2238] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:04.302488 17466 raft_consensus.cc:2267] T c9b3f352d7ae4b1d9a144efaad03a658 P cf15125b30b34f7291283d2ebc3ef969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:04.327790 17466 tablet_server.cc:195] TabletServer@127.17.14.129:0 shutdown complete.
I20250109 21:00:04.468252 21996 consensus_queue.cc:1035] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 21:00:04.487618 21897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250109 21:00:04.491044 21897 consensus_peers.cc:487] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Couldn't send request to peer cf15125b30b34f7291283d2ebc3ef969. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:04.500773 22010 consensus_queue.cc:1035] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 21:00:04.512008 21897 consensus_peers.cc:487] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Couldn't send request to peer cf15125b30b34f7291283d2ebc3ef969. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:04.705650 21996 consensus_queue.cc:1035] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 21:00:04.717048 21897 consensus_peers.cc:487] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Couldn't send request to peer cf15125b30b34f7291283d2ebc3ef969. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:05.107715 22010 consensus_queue.cc:579] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Leader has been unable to successfully communicate with peer cf15125b30b34f7291283d2ebc3ef969 for more than 1 seconds (1.143s)
W20250109 21:00:05.113531 21678 catalog_manager.cc:5204] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7ebe136f04cb4e6eae1fe43a99f49c0e with cas_config_opid_index -1: no extra replica candidate found for tablet 7ebe136f04cb4e6eae1fe43a99f49c0e (table test-workload [id=41198418dc344bd981bbbbbc6a95ad51]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250109 21:00:05.163389 21729 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver cf15125b30b34f7291283d2ebc3ef969 not available for placement
I20250109 21:00:05.210628 22010 consensus_queue.cc:579] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Leader has been unable to successfully communicate with peer cf15125b30b34f7291283d2ebc3ef969 for more than 1 seconds (1.055s)
I20250109 21:00:05.254621 22010 consensus_queue.cc:579] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Leader has been unable to successfully communicate with peer cf15125b30b34f7291283d2ebc3ef969 for more than 1 seconds (1.281s)
I20250109 21:00:05.346973 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:05.353582 22016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:05.354892 22017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:05.357932 17466 server_base.cc:1034] running on GCE node
W20250109 21:00:05.358991 22019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:05.359876 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:05.360100 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:05.360232 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456405360220 us; error 0 us; skew 500 ppm
I20250109 21:00:05.360643 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:05.363233 17466 webserver.cc:458] Webserver started at http://127.17.14.132:37901/ using document root <none> and password file <none>
I20250109 21:00:05.363667 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:05.363847 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:05.364082 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:05.365305 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-3-root/instance:
uuid: "d8594a2254644126895323fc7e75a5b5"
format_stamp: "Formatted at 2025-01-09 21:00:05 on dist-test-slave-j54z"
I20250109 21:00:05.369871 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250109 21:00:05.373085 22024 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:05.373828 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:00:05.374100 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-3-root
uuid: "d8594a2254644126895323fc7e75a5b5"
format_stamp: "Formatted at 2025-01-09 21:00:05 on dist-test-slave-j54z"
I20250109 21:00:05.374356 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736456270445890-17466-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:05.385092 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:05.386292 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:05.387696 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:05.390060 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:00:05.390236 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:05.390477 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:00:05.390627 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:05.429694 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.132:33533
I20250109 21:00:05.429780 22086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.132:33533 every 8 connection(s)
I20250109 21:00:05.444413 22087 heartbeater.cc:346] Connected to a master server at 127.17.14.190:43287
I20250109 21:00:05.444767 22087 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:05.445514 22087 heartbeater.cc:510] Master 127.17.14.190:43287 requested a full tablet report, sending...
I20250109 21:00:05.447234 21678 ts_manager.cc:194] Registered new tserver with Master: d8594a2254644126895323fc7e75a5b5 (127.17.14.132:33533)
I20250109 21:00:05.448719 21678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48178
I20250109 21:00:05.492918 21931 consensus_queue.cc:237] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } }
I20250109 21:00:05.499204 21857 raft_consensus.cc:1270] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:00:05.502707 22010 consensus_queue.cc:1035] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250109 21:00:05.503526 21897 consensus_peers.cc:487] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Couldn't send request to peer cf15125b30b34f7291283d2ebc3ef969. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:05.509994 21996 raft_consensus.cc:2949] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d8594a2254644126895323fc7e75a5b5 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } } }
W20250109 21:00:05.511445 21895 consensus_peers.cc:487] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer d8594a2254644126895323fc7e75a5b5 (127.17.14.132:33533): Couldn't send request to peer d8594a2254644126895323fc7e75a5b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9b3f352d7ae4b1d9a144efaad03a658. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:05.512038 21857 raft_consensus.cc:2949] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d8594a2254644126895323fc7e75a5b5 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } } }
I20250109 21:00:05.520078 21665 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c9b3f352d7ae4b1d9a144efaad03a658 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20250109 21:00:05.524924 21678 catalog_manager.cc:5526] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef reported cstate change: config changed from index -1 to 2, NON_VOTER d8594a2254644126895323fc7e75a5b5 (127.17.14.132) added. New cstate: current_term: 1 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:05.582202 21931 consensus_queue.cc:237] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } }
I20250109 21:00:05.587956 21857 raft_consensus.cc:1270] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250109 21:00:05.588220 21895 consensus_peers.cc:487] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer d8594a2254644126895323fc7e75a5b5 (127.17.14.132:33533): Couldn't send request to peer d8594a2254644126895323fc7e75a5b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 721d29157db9464e96715e12031235e5. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:05.589221 22010 consensus_queue.cc:1035] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250109 21:00:05.590222 21897 consensus_peers.cc:487] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Couldn't send request to peer cf15125b30b34f7291283d2ebc3ef969. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:05.594131 21981 raft_consensus.cc:2949] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d8594a2254644126895323fc7e75a5b5 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } } }
I20250109 21:00:05.595446 21857 raft_consensus.cc:2949] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d8594a2254644126895323fc7e75a5b5 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } } }
I20250109 21:00:05.601202 21665 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 721d29157db9464e96715e12031235e5 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250109 21:00:05.605403 21679 catalog_manager.cc:5526] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef reported cstate change: config changed from index -1 to 2, NON_VOTER d8594a2254644126895323fc7e75a5b5 (127.17.14.132) added. New cstate: current_term: 1 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:05.690070 21931 consensus_queue.cc:237] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } }
W20250109 21:00:05.696089 21895 consensus_peers.cc:487] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer d8594a2254644126895323fc7e75a5b5 (127.17.14.132:33533): Couldn't send request to peer d8594a2254644126895323fc7e75a5b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ebe136f04cb4e6eae1fe43a99f49c0e. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:05.696738 21857 raft_consensus.cc:1270] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:00:05.698021 21981 consensus_queue.cc:1035] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250109 21:00:05.698689 21897 consensus_peers.cc:487] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Couldn't send request to peer cf15125b30b34f7291283d2ebc3ef969. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:05.703611 22010 raft_consensus.cc:2949] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d8594a2254644126895323fc7e75a5b5 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } } }
I20250109 21:00:05.705076 21857 raft_consensus.cc:2949] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d8594a2254644126895323fc7e75a5b5 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } } }
I20250109 21:00:05.713011 21665 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7ebe136f04cb4e6eae1fe43a99f49c0e with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250109 21:00:05.717067 21678 catalog_manager.cc:5526] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef reported cstate change: config changed from index -1 to 2, NON_VOTER d8594a2254644126895323fc7e75a5b5 (127.17.14.132) added. New cstate: current_term: 1 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:05.958851 22095 raft_consensus.cc:491] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:05.959396 22095 raft_consensus.cc:513] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:05.961535 22095 leader_election.cc:290] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149), 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131:39529)
W20250109 21:00:05.967139 21823 leader_election.cc:336] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111)
I20250109 21:00:05.971642 21931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287b207fde4413aae50e6346c490826" candidate_uuid: "15e7316da760420083b0ee12e34ab3f8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" is_pre_election: true
I20250109 21:00:05.972208 21931 raft_consensus.cc:2463] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e7316da760420083b0ee12e34ab3f8 in term 1.
I20250109 21:00:05.973169 21821 leader_election.cc:304] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15e7316da760420083b0ee12e34ab3f8, 77b28859ea234bdfa21ca7e9fe9517ef; no voters: cf15125b30b34f7291283d2ebc3ef969
I20250109 21:00:05.973836 22095 raft_consensus.cc:2798] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250109 21:00:05.974100 22095 raft_consensus.cc:491] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:05.974303 22095 raft_consensus.cc:3054] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:05.978972 22095 raft_consensus.cc:513] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:05.980465 22095 leader_election.cc:290] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [CANDIDATE]: Term 2 election: Requested vote from peers cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149), 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131:39529)
I20250109 21:00:05.981519 21931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287b207fde4413aae50e6346c490826" candidate_uuid: "15e7316da760420083b0ee12e34ab3f8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b28859ea234bdfa21ca7e9fe9517ef"
I20250109 21:00:05.982134 21931 raft_consensus.cc:3054] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:05.984162 22106 ts_tablet_manager.cc:927] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5: Initiating tablet copy from peer 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131:39529)
W20250109 21:00:05.983990 21823 leader_election.cc:336] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111)
I20250109 21:00:05.985818 22106 tablet_copy_client.cc:323] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.131:39529
I20250109 21:00:05.988847 21931 raft_consensus.cc:2463] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e7316da760420083b0ee12e34ab3f8 in term 2.
I20250109 21:00:05.989810 21821 leader_election.cc:304] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15e7316da760420083b0ee12e34ab3f8, 77b28859ea234bdfa21ca7e9fe9517ef; no voters: cf15125b30b34f7291283d2ebc3ef969
I20250109 21:00:05.990633 22095 raft_consensus.cc:2798] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 21:00:05.991763 22095 raft_consensus.cc:695] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 2 LEADER]: Becoming Leader. State: Replica: 15e7316da760420083b0ee12e34ab3f8, State: Running, Role: LEADER
I20250109 21:00:05.992486 22095 consensus_queue.cc:237] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } }
I20250109 21:00:05.996768 21942 tablet_copy_service.cc:140] P 77b28859ea234bdfa21ca7e9fe9517ef: Received BeginTabletCopySession request for tablet c9b3f352d7ae4b1d9a144efaad03a658 from peer d8594a2254644126895323fc7e75a5b5 ({username='slave'} at 127.0.0.1:39202)
I20250109 21:00:05.997262 21942 tablet_copy_service.cc:161] P 77b28859ea234bdfa21ca7e9fe9517ef: Beginning new tablet copy session on tablet c9b3f352d7ae4b1d9a144efaad03a658 from peer d8594a2254644126895323fc7e75a5b5 at {username='slave'} at 127.0.0.1:39202: session id = d8594a2254644126895323fc7e75a5b5-c9b3f352d7ae4b1d9a144efaad03a658
I20250109 21:00:05.999634 21679 catalog_manager.cc:5526] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 reported cstate change: term changed from 1 to 2, leader changed from cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) to 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130). New cstate: current_term: 2 leader_uuid: "15e7316da760420083b0ee12e34ab3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:06.003671 21942 tablet_copy_source_session.cc:215] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 21:00:06.006855 22106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b3f352d7ae4b1d9a144efaad03a658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:06.017329 22106 tablet_copy_client.cc:806] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5: tablet copy: Starting download of 0 data blocks...
I20250109 21:00:06.017841 22106 tablet_copy_client.cc:670] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5: tablet copy: Starting download of 1 WAL segments...
I20250109 21:00:06.020586 22106 tablet_copy_client.cc:538] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 21:00:06.026062 22106 tablet_bootstrap.cc:492] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5: Bootstrap starting.
I20250109 21:00:06.039834 22106 tablet_bootstrap.cc:492] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:06.040468 22106 tablet_bootstrap.cc:492] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5: Bootstrap complete.
I20250109 21:00:06.040977 22106 ts_tablet_manager.cc:1397] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20250109 21:00:06.042937 22106 raft_consensus.cc:357] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } }
I20250109 21:00:06.043495 22106 raft_consensus.cc:738] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8594a2254644126895323fc7e75a5b5, State: Initialized, Role: LEARNER
I20250109 21:00:06.044008 22106 consensus_queue.cc:260] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } }
I20250109 21:00:06.045747 22087 heartbeater.cc:502] Master 127.17.14.190:43287 was elected leader, sending a full tablet report...
I20250109 21:00:06.046104 22106 ts_tablet_manager.cc:1428] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20250109 21:00:06.047761 21942 tablet_copy_service.cc:342] P 77b28859ea234bdfa21ca7e9fe9517ef: Request end of tablet copy session d8594a2254644126895323fc7e75a5b5-c9b3f352d7ae4b1d9a144efaad03a658 received from {username='slave'} at 127.0.0.1:39202
I20250109 21:00:06.048127 21942 tablet_copy_service.cc:434] P 77b28859ea234bdfa21ca7e9fe9517ef: ending tablet copy session d8594a2254644126895323fc7e75a5b5-c9b3f352d7ae4b1d9a144efaad03a658 on tablet c9b3f352d7ae4b1d9a144efaad03a658 with peer d8594a2254644126895323fc7e75a5b5
I20250109 21:00:06.052691 22106 ts_tablet_manager.cc:927] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5: Initiating tablet copy from peer 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131:39529)
I20250109 21:00:06.053946 22106 tablet_copy_client.cc:323] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.131:39529
I20250109 21:00:06.055225 21942 tablet_copy_service.cc:140] P 77b28859ea234bdfa21ca7e9fe9517ef: Received BeginTabletCopySession request for tablet 721d29157db9464e96715e12031235e5 from peer d8594a2254644126895323fc7e75a5b5 ({username='slave'} at 127.0.0.1:39202)
I20250109 21:00:06.055594 21942 tablet_copy_service.cc:161] P 77b28859ea234bdfa21ca7e9fe9517ef: Beginning new tablet copy session on tablet 721d29157db9464e96715e12031235e5 from peer d8594a2254644126895323fc7e75a5b5 at {username='slave'} at 127.0.0.1:39202: session id = d8594a2254644126895323fc7e75a5b5-721d29157db9464e96715e12031235e5
I20250109 21:00:06.060334 21942 tablet_copy_source_session.cc:215] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 21:00:06.062738 22106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 721d29157db9464e96715e12031235e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:06.072472 22106 tablet_copy_client.cc:806] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5: tablet copy: Starting download of 0 data blocks...
I20250109 21:00:06.072945 22106 tablet_copy_client.cc:670] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5: tablet copy: Starting download of 1 WAL segments...
I20250109 21:00:06.075547 22106 tablet_copy_client.cc:538] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 21:00:06.080842 22106 tablet_bootstrap.cc:492] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5: Bootstrap starting.
I20250109 21:00:06.095206 22106 tablet_bootstrap.cc:492] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:06.095806 22106 tablet_bootstrap.cc:492] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5: Bootstrap complete.
I20250109 21:00:06.096263 22106 ts_tablet_manager.cc:1397] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20250109 21:00:06.098115 22106 raft_consensus.cc:357] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } }
I20250109 21:00:06.098655 22106 raft_consensus.cc:738] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8594a2254644126895323fc7e75a5b5, State: Initialized, Role: LEARNER
I20250109 21:00:06.099088 22106 consensus_queue.cc:260] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } }
I20250109 21:00:06.100593 22106 ts_tablet_manager.cc:1428] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250109 21:00:06.102044 21942 tablet_copy_service.cc:342] P 77b28859ea234bdfa21ca7e9fe9517ef: Request end of tablet copy session d8594a2254644126895323fc7e75a5b5-721d29157db9464e96715e12031235e5 received from {username='slave'} at 127.0.0.1:39202
I20250109 21:00:06.102442 21942 tablet_copy_service.cc:434] P 77b28859ea234bdfa21ca7e9fe9517ef: ending tablet copy session d8594a2254644126895323fc7e75a5b5-721d29157db9464e96715e12031235e5 on tablet 721d29157db9464e96715e12031235e5 with peer d8594a2254644126895323fc7e75a5b5
W20250109 21:00:06.165467 21729 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver cf15125b30b34f7291283d2ebc3ef969 not available for placement
I20250109 21:00:06.173298 22106 ts_tablet_manager.cc:927] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5: Initiating tablet copy from peer 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131:39529)
I20250109 21:00:06.174592 22106 tablet_copy_client.cc:323] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.131:39529
I20250109 21:00:06.175884 21942 tablet_copy_service.cc:140] P 77b28859ea234bdfa21ca7e9fe9517ef: Received BeginTabletCopySession request for tablet 7ebe136f04cb4e6eae1fe43a99f49c0e from peer d8594a2254644126895323fc7e75a5b5 ({username='slave'} at 127.0.0.1:39202)
I20250109 21:00:06.176309 21942 tablet_copy_service.cc:161] P 77b28859ea234bdfa21ca7e9fe9517ef: Beginning new tablet copy session on tablet 7ebe136f04cb4e6eae1fe43a99f49c0e from peer d8594a2254644126895323fc7e75a5b5 at {username='slave'} at 127.0.0.1:39202: session id = d8594a2254644126895323fc7e75a5b5-7ebe136f04cb4e6eae1fe43a99f49c0e
I20250109 21:00:06.181061 21942 tablet_copy_source_session.cc:215] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 21:00:06.183209 22106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ebe136f04cb4e6eae1fe43a99f49c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:06.193713 22106 tablet_copy_client.cc:806] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5: tablet copy: Starting download of 0 data blocks...
I20250109 21:00:06.194197 22106 tablet_copy_client.cc:670] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5: tablet copy: Starting download of 1 WAL segments...
I20250109 21:00:06.196897 22106 tablet_copy_client.cc:538] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 21:00:06.202329 22106 tablet_bootstrap.cc:492] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5: Bootstrap starting.
I20250109 21:00:06.214308 22106 tablet_bootstrap.cc:492] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:06.215137 22106 tablet_bootstrap.cc:492] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5: Bootstrap complete.
I20250109 21:00:06.215675 22106 ts_tablet_manager.cc:1397] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250109 21:00:06.217257 22106 raft_consensus.cc:357] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } }
I20250109 21:00:06.217878 22106 raft_consensus.cc:738] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8594a2254644126895323fc7e75a5b5, State: Initialized, Role: LEARNER
I20250109 21:00:06.218302 22106 consensus_queue.cc:260] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } }
I20250109 21:00:06.219626 22106 ts_tablet_manager.cc:1428] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250109 21:00:06.221014 21942 tablet_copy_service.cc:342] P 77b28859ea234bdfa21ca7e9fe9517ef: Request end of tablet copy session d8594a2254644126895323fc7e75a5b5-7ebe136f04cb4e6eae1fe43a99f49c0e received from {username='slave'} at 127.0.0.1:39202
I20250109 21:00:06.221309 21942 tablet_copy_service.cc:434] P 77b28859ea234bdfa21ca7e9fe9517ef: ending tablet copy session d8594a2254644126895323fc7e75a5b5-7ebe136f04cb4e6eae1fe43a99f49c0e on tablet 7ebe136f04cb4e6eae1fe43a99f49c0e with peer d8594a2254644126895323fc7e75a5b5
I20250109 21:00:06.379582 21931 raft_consensus.cc:1270] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Refusing update from remote peer 15e7316da760420083b0ee12e34ab3f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250109 21:00:06.380712 22095 consensus_queue.cc:1035] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 21:00:06.392702 21823 consensus_peers.cc:487] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 -> Peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Couldn't send request to peer cf15125b30b34f7291283d2ebc3ef969. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:06.454629 22062 raft_consensus.cc:1212] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250109 21:00:06.496268 22062 raft_consensus.cc:1212] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250109 21:00:06.646155 22062 raft_consensus.cc:1212] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250109 21:00:06.658381 22122 consensus_queue.cc:579] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Leader has been unable to successfully communicate with peer cf15125b30b34f7291283d2ebc3ef969 for more than 1 seconds (1.164s)
I20250109 21:00:06.739281 22122 consensus_queue.cc:579] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Leader has been unable to successfully communicate with peer cf15125b30b34f7291283d2ebc3ef969 for more than 1 seconds (1.156s)
I20250109 21:00:06.860392 22122 raft_consensus.cc:1059] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef: attempting to promote NON_VOTER d8594a2254644126895323fc7e75a5b5 to VOTER
I20250109 21:00:06.862160 22122 consensus_queue.cc:237] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } }
I20250109 21:00:06.867455 22062 raft_consensus.cc:1270] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5 [term 1 LEARNER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 21:00:06.867596 21857 raft_consensus.cc:1270] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 21:00:06.868692 21996 consensus_queue.cc:1035] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:06.869443 22122 consensus_queue.cc:1035] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250109 21:00:06.870173 21897 consensus_peers.cc:487] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Couldn't send request to peer cf15125b30b34f7291283d2ebc3ef969. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:06.876013 22122 raft_consensus.cc:2949] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.877383 22062 raft_consensus.cc:2949] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.884800 21857 raft_consensus.cc:2949] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.887279 21679 catalog_manager.cc:5526] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef reported cstate change: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250109 21:00:06.890365 22122 raft_consensus.cc:1059] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef: attempting to promote NON_VOTER d8594a2254644126895323fc7e75a5b5 to VOTER
I20250109 21:00:06.893524 22122 consensus_queue.cc:237] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } }
I20250109 21:00:06.910928 22062 raft_consensus.cc:1270] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5 [term 1 LEARNER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 21:00:06.912532 22126 consensus_queue.cc:1035] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250109 21:00:06.914603 21858 raft_consensus.cc:1270] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250109 21:00:06.916800 21897 consensus_peers.cc:487] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Couldn't send request to peer cf15125b30b34f7291283d2ebc3ef969. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:06.919142 22122 consensus_queue.cc:1035] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:06.920431 21931 consensus_queue.cc:237] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } }
I20250109 21:00:06.928100 21858 raft_consensus.cc:1270] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250109 21:00:06.929260 22062 raft_consensus.cc:1270] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250109 21:00:06.930426 22122 consensus_queue.cc:1035] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250109 21:00:06.931420 21996 consensus_queue.cc:1035] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250109 21:00:06.936515 21996 raft_consensus.cc:2949] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.939917 22126 raft_consensus.cc:2949] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.939749 21857 raft_consensus.cc:2949] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.941593 21858 raft_consensus.cc:2949] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.944530 22062 raft_consensus.cc:2949] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.952880 21678 catalog_manager.cc:5526] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 reported cstate change: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.955178 22061 raft_consensus.cc:2949] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.954372 21665 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 721d29157db9464e96715e12031235e5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 21:00:06.961949 21679 catalog_manager.cc:5526] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5 reported cstate change: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New cstate: current_term: 1 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.968789 21932 consensus_queue.cc:237] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } }
I20250109 21:00:06.974679 22061 raft_consensus.cc:1270] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250109 21:00:06.974974 21858 raft_consensus.cc:1270] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250109 21:00:06.976270 22122 consensus_queue.cc:1035] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:06.979156 22122 consensus_queue.cc:1035] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:06.983124 22120 raft_consensus.cc:2949] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.987753 22061 raft_consensus.cc:2949] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.989970 21858 raft_consensus.cc:2949] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:06.992929 21665 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet c9b3f352d7ae4b1d9a144efaad03a658 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250109 21:00:06.996614 21667 catalog_manager.cc:4615] TS cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 721d29157db9464e96715e12031235e5: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111)
I20250109 21:00:07.002676 21678 catalog_manager.cc:5526] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 reported cstate change: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New cstate: current_term: 1 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:07.044008 22122 consensus_queue.cc:579] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Leader has been unable to successfully communicate with peer cf15125b30b34f7291283d2ebc3ef969 for more than 1 seconds (1.353s)
I20250109 21:00:07.108891 22122 raft_consensus.cc:1059] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef: attempting to promote NON_VOTER d8594a2254644126895323fc7e75a5b5 to VOTER
I20250109 21:00:07.110579 22122 consensus_queue.cc:237] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } }
I20250109 21:00:07.115916 22061 raft_consensus.cc:1270] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5 [term 1 LEARNER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 21:00:07.116020 21858 raft_consensus.cc:1270] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 21:00:07.117159 22122 consensus_queue.cc:1035] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:07.117923 22120 consensus_queue.cc:1035] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250109 21:00:07.118810 21897 consensus_peers.cc:487] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Couldn't send request to peer cf15125b30b34f7291283d2ebc3ef969. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:07.125268 22122 raft_consensus.cc:2949] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:07.126881 21858 raft_consensus.cc:2949] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:07.138350 22061 raft_consensus.cc:2949] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:07.138453 21680 catalog_manager.cc:5526] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef reported cstate change: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250109 21:00:07.152199 21932 consensus_queue.cc:237] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } }
I20250109 21:00:07.156628 21858 raft_consensus.cc:1270] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250109 21:00:07.156628 22061 raft_consensus.cc:1270] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250109 21:00:07.158207 22126 consensus_queue.cc:1035] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:07.158897 22120 consensus_queue.cc:1035] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:07.164340 22120 raft_consensus.cc:2949] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:07.166020 21858 raft_consensus.cc:2949] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
W20250109 21:00:07.168808 21729 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver cf15125b30b34f7291283d2ebc3ef969 not available for placement
I20250109 21:00:07.169821 22061 raft_consensus.cc:2949] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:07.172467 21665 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7ebe136f04cb4e6eae1fe43a99f49c0e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 21:00:07.175856 21678 catalog_manager.cc:5526] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef reported cstate change: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New cstate: current_term: 1 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250109 21:00:07.477447 22128 consensus_queue.cc:579] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Leader has been unable to successfully communicate with peer cf15125b30b34f7291283d2ebc3ef969 for more than 1 seconds (1.484s)
I20250109 21:00:07.487380 21858 consensus_queue.cc:237] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } }
I20250109 21:00:07.493388 21932 raft_consensus.cc:1270] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Refusing update from remote peer 15e7316da760420083b0ee12e34ab3f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 21:00:07.500887 22142 consensus_queue.cc:1035] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250109 21:00:07.502573 21823 consensus_peers.cc:487] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 -> Peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Couldn't send request to peer cf15125b30b34f7291283d2ebc3ef969. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:07.508364 22128 raft_consensus.cc:2949] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER d8594a2254644126895323fc7e75a5b5 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } } }
W20250109 21:00:07.509728 21821 consensus_peers.cc:487] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 -> Peer d8594a2254644126895323fc7e75a5b5 (127.17.14.132:33533): Couldn't send request to peer d8594a2254644126895323fc7e75a5b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7287b207fde4413aae50e6346c490826. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:07.509992 21932 raft_consensus.cc:2949] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER d8594a2254644126895323fc7e75a5b5 (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } } }
I20250109 21:00:07.517979 21668 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7287b207fde4413aae50e6346c490826 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250109 21:00:07.523391 21680 catalog_manager.cc:5526] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef reported cstate change: config changed from index -1 to 2, NON_VOTER d8594a2254644126895323fc7e75a5b5 (127.17.14.132) added. New cstate: current_term: 2 leader_uuid: "15e7316da760420083b0ee12e34ab3f8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } } }
I20250109 21:00:08.072774 22150 ts_tablet_manager.cc:927] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5: Initiating tablet copy from peer 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587)
I20250109 21:00:08.074879 22150 tablet_copy_client.cc:323] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.130:38587
I20250109 21:00:08.084389 21868 tablet_copy_service.cc:140] P 15e7316da760420083b0ee12e34ab3f8: Received BeginTabletCopySession request for tablet 7287b207fde4413aae50e6346c490826 from peer d8594a2254644126895323fc7e75a5b5 ({username='slave'} at 127.0.0.1:43764)
I20250109 21:00:08.084815 21868 tablet_copy_service.cc:161] P 15e7316da760420083b0ee12e34ab3f8: Beginning new tablet copy session on tablet 7287b207fde4413aae50e6346c490826 from peer d8594a2254644126895323fc7e75a5b5 at {username='slave'} at 127.0.0.1:43764: session id = d8594a2254644126895323fc7e75a5b5-7287b207fde4413aae50e6346c490826
I20250109 21:00:08.089883 21868 tablet_copy_source_session.cc:215] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 21:00:08.092278 22150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7287b207fde4413aae50e6346c490826. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:08.102489 22150 tablet_copy_client.cc:806] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5: tablet copy: Starting download of 0 data blocks...
I20250109 21:00:08.103018 22150 tablet_copy_client.cc:670] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5: tablet copy: Starting download of 1 WAL segments...
I20250109 21:00:08.106144 22150 tablet_copy_client.cc:538] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 21:00:08.111711 22150 tablet_bootstrap.cc:492] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5: Bootstrap starting.
I20250109 21:00:08.125419 22150 tablet_bootstrap.cc:492] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:08.126125 22150 tablet_bootstrap.cc:492] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5: Bootstrap complete.
I20250109 21:00:08.126617 22150 ts_tablet_manager.cc:1397] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.008s
I20250109 21:00:08.128477 22150 raft_consensus.cc:357] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } }
I20250109 21:00:08.129045 22150 raft_consensus.cc:738] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d8594a2254644126895323fc7e75a5b5, State: Initialized, Role: LEARNER
I20250109 21:00:08.129596 22150 consensus_queue.cc:260] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: true } }
I20250109 21:00:08.132220 22150 ts_tablet_manager.cc:1428] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20250109 21:00:08.133795 21868 tablet_copy_service.cc:342] P 15e7316da760420083b0ee12e34ab3f8: Request end of tablet copy session d8594a2254644126895323fc7e75a5b5-7287b207fde4413aae50e6346c490826 received from {username='slave'} at 127.0.0.1:43764
I20250109 21:00:08.134258 21868 tablet_copy_service.cc:434] P 15e7316da760420083b0ee12e34ab3f8: ending tablet copy session d8594a2254644126895323fc7e75a5b5-7287b207fde4413aae50e6346c490826 on tablet 7287b207fde4413aae50e6346c490826 with peer d8594a2254644126895323fc7e75a5b5
W20250109 21:00:08.137004 21667 catalog_manager.cc:4615] TS cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 721d29157db9464e96715e12031235e5: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111)
W20250109 21:00:08.171994 21729 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver cf15125b30b34f7291283d2ebc3ef969 not available for placement
I20250109 21:00:08.644418 22062 raft_consensus.cc:1212] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250109 21:00:08.833256 22160 consensus_queue.cc:579] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Leader has been unable to successfully communicate with peer cf15125b30b34f7291283d2ebc3ef969 for more than 1 seconds (1.345s)
W20250109 21:00:09.175931 21729 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver cf15125b30b34f7291283d2ebc3ef969 not available for placement
W20250109 21:00:09.176965 21667 catalog_manager.cc:4615] TS cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 721d29157db9464e96715e12031235e5: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111)
I20250109 21:00:09.249065 22160 raft_consensus.cc:1059] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8: attempting to promote NON_VOTER d8594a2254644126895323fc7e75a5b5 to VOTER
I20250109 21:00:09.250941 22160 consensus_queue.cc:237] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } }
I20250109 21:00:09.257320 21932 raft_consensus.cc:1270] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Refusing update from remote peer 15e7316da760420083b0ee12e34ab3f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 21:00:09.257458 22062 raft_consensus.cc:1270] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 2 LEARNER]: Refusing update from remote peer 15e7316da760420083b0ee12e34ab3f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 21:00:09.258684 22166 consensus_queue.cc:1035] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:09.259356 22160 consensus_queue.cc:1035] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250109 21:00:09.260005 21823 consensus_peers.cc:487] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 -> Peer cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): Couldn't send request to peer cf15125b30b34f7291283d2ebc3ef969. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:09.267310 22166 raft_consensus.cc:2949] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:09.268740 22062 raft_consensus.cc:2949] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:09.282121 21932 raft_consensus.cc:2949] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:09.285187 21679 catalog_manager.cc:5526] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 reported cstate change: config changed from index 2 to 3, d8594a2254644126895323fc7e75a5b5 (127.17.14.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "15e7316da760420083b0ee12e34ab3f8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf15125b30b34f7291283d2ebc3ef969" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250109 21:00:09.299986 21858 consensus_queue.cc:237] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } }
I20250109 21:00:09.304208 22062 raft_consensus.cc:1270] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 2 FOLLOWER]: Refusing update from remote peer 15e7316da760420083b0ee12e34ab3f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:09.304343 21932 raft_consensus.cc:1270] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Refusing update from remote peer 15e7316da760420083b0ee12e34ab3f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:09.305338 22142 consensus_queue.cc:1035] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:09.306008 22170 consensus_queue.cc:1035] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:09.311573 22170 raft_consensus.cc:2949] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:09.312845 21932 raft_consensus.cc:2949] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:09.313129 22062 raft_consensus.cc:2949] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } } }
I20250109 21:00:09.321215 21668 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7287b207fde4413aae50e6346c490826 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 21:00:09.324971 21680 catalog_manager.cc:5526] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 reported cstate change: config changed from index 3 to 4, VOTER cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129) evicted. New cstate: current_term: 2 leader_uuid: "15e7316da760420083b0ee12e34ab3f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250109 21:00:09.655011 21667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20250109 21:00:10.503269 21667 catalog_manager.cc:4615] TS cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7287b207fde4413aae50e6346c490826: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111)
W20250109 21:00:11.644850 21667 catalog_manager.cc:4615] TS cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7287b207fde4413aae50e6346c490826: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111)
W20250109 21:00:11.669557 21729 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 77b28859ea234bdfa21ca7e9fe9517ef not available for placement
I20250109 21:00:11.686187 22180 consensus_queue.cc:579] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [LEADER]: Leader has been unable to successfully communicate with peer d8594a2254644126895323fc7e75a5b5 for more than 1 seconds (1.094s)
I20250109 21:00:11.711030 22183 consensus_queue.cc:579] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Leader has been unable to successfully communicate with peer 15e7316da760420083b0ee12e34ab3f8 for more than 1 seconds (1.005s)
W20250109 21:00:12.238350 21712 debug-util.cc:398] Leaking SignalData structure 0x7b080022e2a0 after lost signal to thread 17469
W20250109 21:00:12.239279 21712 debug-util.cc:398] Leaking SignalData structure 0x7b08001d3b80 after lost signal to thread 21715
W20250109 21:00:12.240242 21712 debug-util.cc:398] Leaking SignalData structure 0x7b08001e5a40 after lost signal to thread 21882
W20250109 21:00:12.241075 21712 debug-util.cc:398] Leaking SignalData structure 0x7b080012bee0 after lost signal to thread 21956
W20250109 21:00:12.241997 21712 debug-util.cc:398] Leaking SignalData structure 0x7b08001de4a0 after lost signal to thread 22086
I20250109 21:00:13.456126 17466 tablet_server.cc:178] TabletServer@127.17.14.130:0 shutting down...
I20250109 21:00:13.483578 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:13.484403 17466 tablet_replica.cc:331] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8: stopping tablet replica
I20250109 21:00:13.485231 17466 raft_consensus.cc:2238] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 2 LEADER]: Raft consensus shutting down.
I20250109 21:00:13.486306 17466 raft_consensus.cc:2267] T 7287b207fde4413aae50e6346c490826 P 15e7316da760420083b0ee12e34ab3f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:13.489048 17466 tablet_replica.cc:331] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8: stopping tablet replica
I20250109 21:00:13.489812 17466 raft_consensus.cc:2238] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:13.490573 17466 raft_consensus.cc:2267] T 721d29157db9464e96715e12031235e5 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:13.493067 17466 tablet_replica.cc:331] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8: stopping tablet replica
I20250109 21:00:13.493799 17466 raft_consensus.cc:2238] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:13.494556 17466 raft_consensus.cc:2267] T c9b3f352d7ae4b1d9a144efaad03a658 P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:13.497066 17466 tablet_replica.cc:331] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8: stopping tablet replica
I20250109 21:00:13.497735 17466 raft_consensus.cc:2238] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:13.498463 17466 raft_consensus.cc:2267] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 15e7316da760420083b0ee12e34ab3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 21:00:13.659807 21898 consensus_peers.cc:487] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587): Couldn't send request to peer 15e7316da760420083b0ee12e34ab3f8. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.130:38587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250109 21:00:13.695310 21898 consensus_peers.cc:487] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587): Couldn't send request to peer 15e7316da760420083b0ee12e34ab3f8. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.130:38587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250109 21:00:13.744830 21667 catalog_manager.cc:4615] TS cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7287b207fde4413aae50e6346c490826: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111)
W20250109 21:00:13.831348 21898 consensus_peers.cc:487] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587): Couldn't send request to peer 15e7316da760420083b0ee12e34ab3f8. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.130:38587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250109 21:00:14.158762 21879 debug-util.cc:398] Leaking SignalData structure 0x7b08001d35a0 after lost signal to thread 17469
W20250109 21:00:14.162040 21879 debug-util.cc:398] Leaking SignalData structure 0x7b08000dae20 after lost signal to thread 21715
W20250109 21:00:14.164896 21879 debug-util.cc:398] Leaking SignalData structure 0x7b08001a1ac0 after lost signal to thread 21882
W20250109 21:00:14.167485 21879 debug-util.cc:398] Leaking SignalData structure 0x7b0800139180 after lost signal to thread 21956
W20250109 21:00:14.170185 21879 debug-util.cc:398] Leaking SignalData structure 0x7b08001646c0 after lost signal to thread 22086
I20250109 21:00:14.326985 22194 consensus_queue.cc:579] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Leader has been unable to successfully communicate with peer 15e7316da760420083b0ee12e34ab3f8 for more than 1 seconds (1.113s)
W20250109 21:00:14.523928 17466 thread.cc:535] Waited for 1000ms trying to join with diag-logger (tid 21879)
I20250109 21:00:14.604387 22192 consensus_queue.cc:579] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Leader has been unable to successfully communicate with peer 15e7316da760420083b0ee12e34ab3f8 for more than 1 seconds (1.411s)
I20250109 21:00:14.684698 22195 raft_consensus.cc:491] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 15e7316da760420083b0ee12e34ab3f8)
I20250109 21:00:14.685767 22195 raft_consensus.cc:513] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } }
W20250109 21:00:14.688642 21729 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 15e7316da760420083b0ee12e34ab3f8 not available for placement
I20250109 21:00:14.690557 22195 leader_election.cc:290] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587), 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131:39529)
I20250109 21:00:14.691892 21932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287b207fde4413aae50e6346c490826" candidate_uuid: "d8594a2254644126895323fc7e75a5b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" is_pre_election: true
W20250109 21:00:14.697852 22028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.14.130:38587: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20250109 21:00:14.701884 22028 leader_election.cc:336] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587): Network error: Client connection negotiation failed: client connection to 127.17.14.130:38587: connect: Connection refused (error 111)
I20250109 21:00:14.702487 22028 leader_election.cc:304] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8594a2254644126895323fc7e75a5b5; no voters: 15e7316da760420083b0ee12e34ab3f8, 77b28859ea234bdfa21ca7e9fe9517ef
I20250109 21:00:14.703428 22195 raft_consensus.cc:2743] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250109 21:00:14.709245 22194 raft_consensus.cc:491] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 15e7316da760420083b0ee12e34ab3f8)
I20250109 21:00:14.709998 22194 raft_consensus.cc:513] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } }
I20250109 21:00:14.712394 22194 leader_election.cc:290] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587), d8594a2254644126895323fc7e75a5b5 (127.17.14.132:33533)
I20250109 21:00:14.714305 22061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287b207fde4413aae50e6346c490826" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8594a2254644126895323fc7e75a5b5" is_pre_election: true
I20250109 21:00:14.715051 22061 raft_consensus.cc:2463] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b28859ea234bdfa21ca7e9fe9517ef in term 2.
I20250109 21:00:14.716559 21895 leader_election.cc:304] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77b28859ea234bdfa21ca7e9fe9517ef, d8594a2254644126895323fc7e75a5b5; no voters:
W20250109 21:00:14.717388 21898 leader_election.cc:336] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587): Network error: Client connection negotiation failed: client connection to 127.17.14.130:38587: connect: Connection refused (error 111)
I20250109 21:00:14.717628 22194 raft_consensus.cc:2798] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250109 21:00:14.718050 22194 raft_consensus.cc:491] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Starting leader election (detected failure of leader 15e7316da760420083b0ee12e34ab3f8)
I20250109 21:00:14.718420 22194 raft_consensus.cc:3054] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 2 FOLLOWER]: Advancing to term 3
I20250109 21:00:14.728263 22194 raft_consensus.cc:513] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } }
I20250109 21:00:14.731688 22061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287b207fde4413aae50e6346c490826" candidate_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8594a2254644126895323fc7e75a5b5"
I20250109 21:00:14.732383 22061 raft_consensus.cc:3054] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 2 FOLLOWER]: Advancing to term 3
W20250109 21:00:14.735527 21898 leader_election.cc:336] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587): Network error: Client connection negotiation failed: client connection to 127.17.14.130:38587: connect: Connection refused (error 111)
I20250109 21:00:14.737133 22194 leader_election.cc:290] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 3 election: Requested vote from peers 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587), d8594a2254644126895323fc7e75a5b5 (127.17.14.132:33533)
I20250109 21:00:14.741395 22061 raft_consensus.cc:2463] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b28859ea234bdfa21ca7e9fe9517ef in term 3.
I20250109 21:00:14.742834 21895 leader_election.cc:304] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77b28859ea234bdfa21ca7e9fe9517ef, d8594a2254644126895323fc7e75a5b5; no voters: 15e7316da760420083b0ee12e34ab3f8
I20250109 21:00:14.743821 22194 raft_consensus.cc:2798] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 3 FOLLOWER]: Leader election won for term 3
I20250109 21:00:14.744318 22194 raft_consensus.cc:695] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 3 LEADER]: Becoming Leader. State: Replica: 77b28859ea234bdfa21ca7e9fe9517ef, State: Running, Role: LEADER
I20250109 21:00:14.745467 22194 consensus_queue.cc:237] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } }
I20250109 21:00:14.762074 21678 catalog_manager.cc:5526] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef reported cstate change: term changed from 2 to 3, leader changed from 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130) to 77b28859ea234bdfa21ca7e9fe9517ef (127.17.14.131). New cstate: current_term: 3 leader_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15e7316da760420083b0ee12e34ab3f8" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b28859ea234bdfa21ca7e9fe9517ef" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:14.792480 22194 consensus_queue.cc:579] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Leader has been unable to successfully communicate with peer 15e7316da760420083b0ee12e34ab3f8 for more than 1 seconds (1.537s)
I20250109 21:00:15.252705 22061 raft_consensus.cc:1270] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 3 FOLLOWER]: Refusing update from remote peer 77b28859ea234bdfa21ca7e9fe9517ef: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250109 21:00:15.254287 22192 consensus_queue.cc:1035] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8594a2254644126895323fc7e75a5b5" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 33533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250109 21:00:15.272887 21898 consensus_peers.cc:487] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587): Couldn't send request to peer 15e7316da760420083b0ee12e34ab3f8. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.130:38587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250109 21:00:15.375772 21667 catalog_manager.cc:4615] TS cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 721d29157db9464e96715e12031235e5: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111)
W20250109 21:00:15.524530 17466 thread.cc:535] Waited for 2000ms trying to join with diag-logger (tid 21879)
W20250109 21:00:15.690994 21729 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 15e7316da760420083b0ee12e34ab3f8 not available for placement
W20250109 21:00:16.152846 21898 consensus_peers.cc:487] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587): Couldn't send request to peer 15e7316da760420083b0ee12e34ab3f8. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.130:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250109 21:00:16.165033 22209 consensus_queue.cc:579] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [LEADER]: Leader has been unable to successfully communicate with peer 15e7316da760420083b0ee12e34ab3f8 for more than 1 seconds (1.418s)
W20250109 21:00:16.175463 21898 consensus_peers.cc:487] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587): Couldn't send request to peer 15e7316da760420083b0ee12e34ab3f8. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.130:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250109 21:00:16.373227 21898 consensus_peers.cc:487] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef -> Peer 15e7316da760420083b0ee12e34ab3f8 (127.17.14.130:38587): Couldn't send request to peer 15e7316da760420083b0ee12e34ab3f8. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.130:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250109 21:00:16.524966 17466 thread.cc:535] Waited for 3000ms trying to join with diag-logger (tid 21879)
W20250109 21:00:16.693104 21729 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 15e7316da760420083b0ee12e34ab3f8 not available for placement
I20250109 21:00:16.995724 17466 tablet_server.cc:195] TabletServer@127.17.14.130:0 shutdown complete.
I20250109 21:00:17.014619 17466 tablet_server.cc:178] TabletServer@127.17.14.131:0 shutting down...
I20250109 21:00:17.035010 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:17.035722 17466 tablet_replica.cc:331] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef: stopping tablet replica
I20250109 21:00:17.036320 17466 raft_consensus.cc:2238] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 3 LEADER]: Raft consensus shutting down.
I20250109 21:00:17.037470 17466 raft_consensus.cc:2267] T 7287b207fde4413aae50e6346c490826 P 77b28859ea234bdfa21ca7e9fe9517ef [term 3 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:17.039465 17466 tablet_replica.cc:331] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef: stopping tablet replica
I20250109 21:00:17.039973 17466 raft_consensus.cc:2238] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:17.040753 17466 raft_consensus.cc:2267] T 721d29157db9464e96715e12031235e5 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:17.042773 17466 tablet_replica.cc:331] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef: stopping tablet replica
I20250109 21:00:17.043251 17466 raft_consensus.cc:2238] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:17.043977 17466 raft_consensus.cc:2267] T 7ebe136f04cb4e6eae1fe43a99f49c0e P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:17.045866 17466 tablet_replica.cc:331] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef: stopping tablet replica
I20250109 21:00:17.046309 17466 raft_consensus.cc:2238] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:17.047029 17466 raft_consensus.cc:2267] T c9b3f352d7ae4b1d9a144efaad03a658 P 77b28859ea234bdfa21ca7e9fe9517ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:17.069514 17466 tablet_server.cc:195] TabletServer@127.17.14.131:0 shutdown complete.
I20250109 21:00:17.084578 17466 tablet_server.cc:178] TabletServer@127.17.14.132:0 shutting down...
I20250109 21:00:17.106441 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:17.107142 17466 tablet_replica.cc:331] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5: stopping tablet replica
I20250109 21:00:17.107769 17466 raft_consensus.cc:2238] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:17.108464 17466 raft_consensus.cc:2267] T 7287b207fde4413aae50e6346c490826 P d8594a2254644126895323fc7e75a5b5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:17.110484 17466 tablet_replica.cc:331] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5: stopping tablet replica
I20250109 21:00:17.110991 17466 raft_consensus.cc:2238] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:17.111622 17466 raft_consensus.cc:2267] T 721d29157db9464e96715e12031235e5 P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:17.113473 17466 tablet_replica.cc:331] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5: stopping tablet replica
I20250109 21:00:17.113966 17466 raft_consensus.cc:2238] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:17.114617 17466 raft_consensus.cc:2267] T 7ebe136f04cb4e6eae1fe43a99f49c0e P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:17.116431 17466 tablet_replica.cc:331] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5: stopping tablet replica
I20250109 21:00:17.116858 17466 raft_consensus.cc:2238] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:17.117445 17466 raft_consensus.cc:2267] T c9b3f352d7ae4b1d9a144efaad03a658 P d8594a2254644126895323fc7e75a5b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:17.139710 17466 tablet_server.cc:195] TabletServer@127.17.14.132:0 shutdown complete.
I20250109 21:00:17.153662 17466 master.cc:537] Master@127.17.14.190:43287 shutting down...
W20250109 21:00:17.848963 21667 catalog_manager.cc:4615] TS cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7287b207fde4413aae50e6346c490826: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111)
W20250109 21:00:23.583575 21667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111) [suppressed 25 similar messages]
W20250109 21:00:23.586269 21667 catalog_manager.cc:4615] TS cf15125b30b34f7291283d2ebc3ef969 (127.17.14.129:34149): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 721d29157db9464e96715e12031235e5: Network error: Client connection negotiation failed: client connection to 127.17.14.129:34149: connect: Connection refused (error 111)
I20250109 21:00:24.158754 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:24.159281 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:24.159598 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P ff989a757ef74f3fa22bb4354b0fd85f: stopping tablet replica
I20250109 21:00:24.208801 17466 master.cc:559] Master@127.17.14.190:43287 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (22218 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20250109 21:00:24.256053 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:39061
I20250109 21:00:24.257133 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:24.262701 22216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:24.263098 22217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:24.263919 22219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:24.264267 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:24.265154 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:24.265352 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:24.265470 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456424265460 us; error 0 us; skew 500 ppm
I20250109 21:00:24.265967 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:24.268357 17466 webserver.cc:458] Webserver started at http://127.17.14.190:45095/ using document root <none> and password file <none>
I20250109 21:00:24.268823 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:24.268982 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:24.269207 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:24.270648 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "f6da49b7dc424a49aebed70a9e2d8c68"
format_stamp: "Formatted at 2025-01-09 21:00:24 on dist-test-slave-j54z"
I20250109 21:00:24.275233 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20250109 21:00:24.278414 22224 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:24.279114 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:00:24.279362 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "f6da49b7dc424a49aebed70a9e2d8c68"
format_stamp: "Formatted at 2025-01-09 21:00:24 on dist-test-slave-j54z"
I20250109 21:00:24.279628 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:24.307320 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:24.308511 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:24.347397 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:39061
I20250109 21:00:24.347489 22275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:39061 every 8 connection(s)
I20250109 21:00:24.351316 22276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:24.362764 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68: Bootstrap starting.
I20250109 21:00:24.367286 22276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:24.371302 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68: No bootstrap required, opened a new log
I20250109 21:00:24.373222 22276 raft_consensus.cc:357] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6da49b7dc424a49aebed70a9e2d8c68" member_type: VOTER }
I20250109 21:00:24.373732 22276 raft_consensus.cc:383] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:24.373914 22276 raft_consensus.cc:738] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6da49b7dc424a49aebed70a9e2d8c68, State: Initialized, Role: FOLLOWER
I20250109 21:00:24.374351 22276 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6da49b7dc424a49aebed70a9e2d8c68" member_type: VOTER }
I20250109 21:00:24.374742 22276 raft_consensus.cc:397] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:00:24.374910 22276 raft_consensus.cc:491] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:00:24.375123 22276 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:24.379563 22276 raft_consensus.cc:513] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6da49b7dc424a49aebed70a9e2d8c68" member_type: VOTER }
I20250109 21:00:24.380007 22276 leader_election.cc:304] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6da49b7dc424a49aebed70a9e2d8c68; no voters:
I20250109 21:00:24.381067 22276 leader_election.cc:290] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:00:24.381458 22279 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:24.382809 22279 raft_consensus.cc:695] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [term 1 LEADER]: Becoming Leader. State: Replica: f6da49b7dc424a49aebed70a9e2d8c68, State: Running, Role: LEADER
I20250109 21:00:24.383376 22279 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6da49b7dc424a49aebed70a9e2d8c68" member_type: VOTER }
I20250109 21:00:24.384002 22276 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:00:24.389350 22280 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6da49b7dc424a49aebed70a9e2d8c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6da49b7dc424a49aebed70a9e2d8c68" member_type: VOTER } }
I20250109 21:00:24.389822 22280 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [sys.catalog]: This master's current role is: LEADER
I20250109 21:00:24.390867 22281 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6da49b7dc424a49aebed70a9e2d8c68. Latest consensus state: current_term: 1 leader_uuid: "f6da49b7dc424a49aebed70a9e2d8c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6da49b7dc424a49aebed70a9e2d8c68" member_type: VOTER } }
I20250109 21:00:24.391477 22286 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:00:24.391563 22281 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [sys.catalog]: This master's current role is: LEADER
I20250109 21:00:24.396795 22286 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:00:24.399386 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 21:00:24.405305 22286 catalog_manager.cc:1348] Generated new cluster ID: b843ddac460b47aa900130282e2f555f
I20250109 21:00:24.405591 22286 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:00:24.431167 22286 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:00:24.432421 22286 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:00:24.445587 22286 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68: Generated new TSK 0
I20250109 21:00:24.446141 22286 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:00:24.466382 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:24.471971 22297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:24.473059 22298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:24.474367 17466 server_base.cc:1034] running on GCE node
W20250109 21:00:24.474747 22300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:24.475535 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:24.475739 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:24.475888 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456424475870 us; error 0 us; skew 500 ppm
I20250109 21:00:24.476388 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:24.478581 17466 webserver.cc:458] Webserver started at http://127.17.14.129:44681/ using document root <none> and password file <none>
I20250109 21:00:24.479051 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:24.479226 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:24.479480 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:24.480753 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-0-root/instance:
uuid: "06d140c631d74fe99a0d50ffd8a7fad1"
format_stamp: "Formatted at 2025-01-09 21:00:24 on dist-test-slave-j54z"
I20250109 21:00:24.485224 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 21:00:24.488382 22305 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:24.489089 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:00:24.489332 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "06d140c631d74fe99a0d50ffd8a7fad1"
format_stamp: "Formatted at 2025-01-09 21:00:24 on dist-test-slave-j54z"
I20250109 21:00:24.489660 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:24.512530 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:24.513677 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:24.515044 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:24.517271 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:00:24.517457 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:24.517704 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:00:24.517851 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:24.555936 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:44979
I20250109 21:00:24.556032 22367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:44979 every 8 connection(s)
I20250109 21:00:24.560716 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:24.568230 22372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:24.569926 22373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:24.574590 22368 heartbeater.cc:346] Connected to a master server at 127.17.14.190:39061
W20250109 21:00:24.574769 22375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:24.574983 22368 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:24.575788 22368 heartbeater.cc:510] Master 127.17.14.190:39061 requested a full tablet report, sending...
I20250109 21:00:24.576329 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:24.577435 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:24.577690 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:24.577848 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456424577832 us; error 0 us; skew 500 ppm
I20250109 21:00:24.577821 22241 ts_manager.cc:194] Registered new tserver with Master: 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979)
I20250109 21:00:24.578574 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:24.579902 22241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52470
I20250109 21:00:24.581139 17466 webserver.cc:458] Webserver started at http://127.17.14.130:37469/ using document root <none> and password file <none>
I20250109 21:00:24.581797 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:24.582013 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:24.582293 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:24.583407 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-1-root/instance:
uuid: "02ad93d307eb412b9489918537829a87"
format_stamp: "Formatted at 2025-01-09 21:00:24 on dist-test-slave-j54z"
I20250109 21:00:24.587531 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250109 21:00:24.590512 22380 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:24.591287 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:00:24.591573 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "02ad93d307eb412b9489918537829a87"
format_stamp: "Formatted at 2025-01-09 21:00:24 on dist-test-slave-j54z"
I20250109 21:00:24.591820 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:24.614157 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:24.615270 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:24.616603 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:24.618741 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:00:24.618906 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:24.619124 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:00:24.619256 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:24.656569 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:39015
I20250109 21:00:24.656669 22442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:39015 every 8 connection(s)
I20250109 21:00:24.661212 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:24.668586 22446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:24.669843 22447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:24.670614 22449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:24.671388 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:24.672372 22443 heartbeater.cc:346] Connected to a master server at 127.17.14.190:39061
I20250109 21:00:24.672421 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:24.672696 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:24.672751 22443 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:24.672912 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456424672893 us; error 0 us; skew 500 ppm
I20250109 21:00:24.673547 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:24.673604 22443 heartbeater.cc:510] Master 127.17.14.190:39061 requested a full tablet report, sending...
I20250109 21:00:24.675876 22241 ts_manager.cc:194] Registered new tserver with Master: 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015)
I20250109 21:00:24.676124 17466 webserver.cc:458] Webserver started at http://127.17.14.131:46833/ using document root <none> and password file <none>
I20250109 21:00:24.676745 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:24.676980 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:24.677290 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:24.677702 22241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52480
I20250109 21:00:24.678823 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-2-root/instance:
uuid: "6a8775de2707400c87d2357cb2c09e97"
format_stamp: "Formatted at 2025-01-09 21:00:24 on dist-test-slave-j54z"
I20250109 21:00:24.683079 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250109 21:00:24.686172 22454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:24.686877 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:00:24.687129 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "6a8775de2707400c87d2357cb2c09e97"
format_stamp: "Formatted at 2025-01-09 21:00:24 on dist-test-slave-j54z"
I20250109 21:00:24.687390 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:24.699890 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:24.701035 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:24.702440 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:24.704651 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:00:24.704836 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:24.705041 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:00:24.705188 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:24.743186 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:44163
I20250109 21:00:24.743302 22516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:44163 every 8 connection(s)
I20250109 21:00:24.755788 22517 heartbeater.cc:346] Connected to a master server at 127.17.14.190:39061
I20250109 21:00:24.756129 22517 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:24.756801 22517 heartbeater.cc:510] Master 127.17.14.190:39061 requested a full tablet report, sending...
I20250109 21:00:24.758602 22241 ts_manager.cc:194] Registered new tserver with Master: 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:24.759476 17466 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013165088s
I20250109 21:00:24.759904 22241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52492
I20250109 21:00:25.582702 22368 heartbeater.cc:502] Master 127.17.14.190:39061 was elected leader, sending a full tablet report...
I20250109 21:00:25.680104 22443 heartbeater.cc:502] Master 127.17.14.190:39061 was elected leader, sending a full tablet report...
I20250109 21:00:25.762523 22517 heartbeater.cc:502] Master 127.17.14.190:39061 was elected leader, sending a full tablet report...
I20250109 21:00:25.791543 17466 test_util.cc:274] Using random seed: 1148018719
I20250109 21:00:25.812636 22241 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52508:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:00:25.814885 22241 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:00:25.875631 22482 tablet_service.cc:1467] Processing CreateTablet for tablet 76c3bf7218af4369bec2bdc85e3b5532 (DEFAULT_TABLE table=test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 21:00:25.877079 22482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c3bf7218af4369bec2bdc85e3b5532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:25.877280 22481 tablet_service.cc:1467] Processing CreateTablet for tablet f9ff955074da4768beb84be0b6f54034 (DEFAULT_TABLE table=test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250109 21:00:25.878582 22481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9ff955074da4768beb84be0b6f54034. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:25.882648 22480 tablet_service.cc:1467] Processing CreateTablet for tablet 7d9aa6c74de24f019423b53c76147569 (DEFAULT_TABLE table=test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 21:00:25.883932 22480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9aa6c74de24f019423b53c76147569. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:25.884829 22479 tablet_service.cc:1467] Processing CreateTablet for tablet f400fe6396994714a222603d757ed895 (DEFAULT_TABLE table=test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 21:00:25.886145 22479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f400fe6396994714a222603d757ed895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:25.895715 22408 tablet_service.cc:1467] Processing CreateTablet for tablet 76c3bf7218af4369bec2bdc85e3b5532 (DEFAULT_TABLE table=test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 21:00:25.897034 22408 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c3bf7218af4369bec2bdc85e3b5532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:25.897964 22333 tablet_service.cc:1467] Processing CreateTablet for tablet 76c3bf7218af4369bec2bdc85e3b5532 (DEFAULT_TABLE table=test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 21:00:25.898654 22407 tablet_service.cc:1467] Processing CreateTablet for tablet f9ff955074da4768beb84be0b6f54034 (DEFAULT_TABLE table=test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250109 21:00:25.899281 22333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c3bf7218af4369bec2bdc85e3b5532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:25.899906 22407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9ff955074da4768beb84be0b6f54034. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:25.902608 22406 tablet_service.cc:1467] Processing CreateTablet for tablet 7d9aa6c74de24f019423b53c76147569 (DEFAULT_TABLE table=test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 21:00:25.903882 22406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9aa6c74de24f019423b53c76147569. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:25.906766 22405 tablet_service.cc:1467] Processing CreateTablet for tablet f400fe6396994714a222603d757ed895 (DEFAULT_TABLE table=test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 21:00:25.908326 22405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f400fe6396994714a222603d757ed895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:25.908159 22332 tablet_service.cc:1467] Processing CreateTablet for tablet f9ff955074da4768beb84be0b6f54034 (DEFAULT_TABLE table=test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250109 21:00:25.909757 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9ff955074da4768beb84be0b6f54034. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:25.911115 22331 tablet_service.cc:1467] Processing CreateTablet for tablet 7d9aa6c74de24f019423b53c76147569 (DEFAULT_TABLE table=test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 21:00:25.912904 22330 tablet_service.cc:1467] Processing CreateTablet for tablet f400fe6396994714a222603d757ed895 (DEFAULT_TABLE table=test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 21:00:25.914156 22330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f400fe6396994714a222603d757ed895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:25.915721 22331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9aa6c74de24f019423b53c76147569. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:25.948339 22539 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap starting.
I20250109 21:00:25.950506 22537 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap starting.
I20250109 21:00:25.954777 22539 tablet_bootstrap.cc:654] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:25.957319 22537 tablet_bootstrap.cc:654] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:25.958223 22538 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87: Bootstrap starting.
I20250109 21:00:25.964195 22538 tablet_bootstrap.cc:654] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:25.966630 22537 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: No bootstrap required, opened a new log
I20250109 21:00:25.967098 22539 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: No bootstrap required, opened a new log
I20250109 21:00:25.967157 22537 ts_tablet_manager.cc:1397] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.009s
I20250109 21:00:25.967748 22539 ts_tablet_manager.cc:1397] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.006s
I20250109 21:00:25.968995 22538 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87: No bootstrap required, opened a new log
I20250109 21:00:25.969385 22538 ts_tablet_manager.cc:1397] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20250109 21:00:25.970157 22537 raft_consensus.cc:357] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:25.970350 22539 raft_consensus.cc:357] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:25.970979 22537 raft_consensus.cc:383] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:25.971112 22539 raft_consensus.cc:383] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:25.971311 22537 raft_consensus.cc:738] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8775de2707400c87d2357cb2c09e97, State: Initialized, Role: FOLLOWER
I20250109 21:00:25.971436 22539 raft_consensus.cc:738] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d140c631d74fe99a0d50ffd8a7fad1, State: Initialized, Role: FOLLOWER
I20250109 21:00:25.971418 22538 raft_consensus.cc:357] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:25.972342 22538 raft_consensus.cc:383] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:25.972057 22537 consensus_queue.cc:260] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:25.972616 22538 raft_consensus.cc:738] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ad93d307eb412b9489918537829a87, State: Initialized, Role: FOLLOWER
I20250109 21:00:25.972406 22539 consensus_queue.cc:260] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:25.973270 22538 consensus_queue.cc:260] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:25.977802 22537 ts_tablet_manager.cc:1428] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20250109 21:00:25.976347 22538 ts_tablet_manager.cc:1428] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20250109 21:00:25.978859 22537 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap starting.
I20250109 21:00:25.984158 22539 ts_tablet_manager.cc:1428] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent starting tablet: real 0.016s user 0.007s sys 0.004s
I20250109 21:00:25.988493 22537 tablet_bootstrap.cc:654] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:25.985226 22539 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap starting.
I20250109 21:00:25.990105 22538 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: Bootstrap starting.
I20250109 21:00:25.996127 22539 tablet_bootstrap.cc:654] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:25.996619 22538 tablet_bootstrap.cc:654] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:25.997077 22537 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97: No bootstrap required, opened a new log
I20250109 21:00:25.997620 22537 ts_tablet_manager.cc:1397] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
W20250109 21:00:25.998112 22518 tablet.cc:2367] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:00:26.000710 22537 raft_consensus.cc:357] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.001230 22537 raft_consensus.cc:383] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:26.001498 22537 raft_consensus.cc:738] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8775de2707400c87d2357cb2c09e97, State: Initialized, Role: FOLLOWER
I20250109 21:00:26.001689 22539 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: No bootstrap required, opened a new log
I20250109 21:00:26.002012 22538 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: No bootstrap required, opened a new log
I20250109 21:00:26.002188 22539 ts_tablet_manager.cc:1397] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.003s
I20250109 21:00:26.002467 22538 ts_tablet_manager.cc:1397] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20250109 21:00:26.002120 22537 consensus_queue.cc:260] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.004233 22537 ts_tablet_manager.cc:1428] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:00:26.005125 22537 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap starting.
I20250109 21:00:26.005101 22539 raft_consensus.cc:357] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.005338 22538 raft_consensus.cc:357] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.006096 22539 raft_consensus.cc:383] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:26.006196 22538 raft_consensus.cc:383] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:26.006497 22539 raft_consensus.cc:738] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d140c631d74fe99a0d50ffd8a7fad1, State: Initialized, Role: FOLLOWER
I20250109 21:00:26.006587 22538 raft_consensus.cc:738] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ad93d307eb412b9489918537829a87, State: Initialized, Role: FOLLOWER
I20250109 21:00:26.007351 22539 consensus_queue.cc:260] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.007395 22538 consensus_queue.cc:260] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.009842 22539 ts_tablet_manager.cc:1428] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250109 21:00:26.010869 22539 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap starting.
I20250109 21:00:26.014951 22537 tablet_bootstrap.cc:654] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:26.016816 22539 tablet_bootstrap.cc:654] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:26.016445 22538 ts_tablet_manager.cc:1428] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20250109 21:00:26.026010 22538 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: Bootstrap starting.
I20250109 21:00:26.033548 22539 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1: No bootstrap required, opened a new log
I20250109 21:00:26.033700 22538 tablet_bootstrap.cc:654] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:26.033706 22537 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97: No bootstrap required, opened a new log
I20250109 21:00:26.033984 22539 ts_tablet_manager.cc:1397] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.012s
I20250109 21:00:26.034451 22537 ts_tablet_manager.cc:1397] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.007s
I20250109 21:00:26.036207 22539 raft_consensus.cc:357] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.036759 22539 raft_consensus.cc:383] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:26.036984 22539 raft_consensus.cc:738] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d140c631d74fe99a0d50ffd8a7fad1, State: Initialized, Role: FOLLOWER
I20250109 21:00:26.037417 22537 raft_consensus.cc:357] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.038035 22537 raft_consensus.cc:383] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:26.037760 22539 consensus_queue.cc:260] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.038331 22537 raft_consensus.cc:738] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8775de2707400c87d2357cb2c09e97, State: Initialized, Role: FOLLOWER
I20250109 21:00:26.039649 22538 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: No bootstrap required, opened a new log
I20250109 21:00:26.039184 22537 consensus_queue.cc:260] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.040125 22539 ts_tablet_manager.cc:1428] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250109 21:00:26.040153 22538 ts_tablet_manager.cc:1397] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.005s
I20250109 21:00:26.041306 22539 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap starting.
I20250109 21:00:26.043221 22538 raft_consensus.cc:357] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.043941 22538 raft_consensus.cc:383] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:26.044230 22538 raft_consensus.cc:738] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ad93d307eb412b9489918537829a87, State: Initialized, Role: FOLLOWER
I20250109 21:00:26.045061 22537 ts_tablet_manager.cc:1428] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250109 21:00:26.044972 22538 consensus_queue.cc:260] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.046121 22537 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap starting.
I20250109 21:00:26.047854 22539 tablet_bootstrap.cc:654] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:26.049854 22545 raft_consensus.cc:491] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:26.049978 22538 ts_tablet_manager.cc:1428] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20250109 21:00:26.050330 22545 raft_consensus.cc:513] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.051019 22538 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: Bootstrap starting.
I20250109 21:00:26.052610 22537 tablet_bootstrap.cc:654] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:26.058264 22538 tablet_bootstrap.cc:654] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:26.064854 22537 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97: No bootstrap required, opened a new log
I20250109 21:00:26.065536 22537 ts_tablet_manager.cc:1397] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.011s
I20250109 21:00:26.068336 22537 raft_consensus.cc:357] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.069144 22537 raft_consensus.cc:383] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:26.069504 22537 raft_consensus.cc:738] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8775de2707400c87d2357cb2c09e97, State: Initialized, Role: FOLLOWER
I20250109 21:00:26.070371 22537 consensus_queue.cc:260] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.071319 22538 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: No bootstrap required, opened a new log
I20250109 21:00:26.071882 22538 ts_tablet_manager.cc:1397] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.003s
I20250109 21:00:26.073006 22537 ts_tablet_manager.cc:1428] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250109 21:00:26.074447 22538 raft_consensus.cc:357] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.074975 22538 raft_consensus.cc:383] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:26.075423 22538 raft_consensus.cc:738] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ad93d307eb412b9489918537829a87, State: Initialized, Role: FOLLOWER
I20250109 21:00:26.075474 22543 raft_consensus.cc:491] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:26.076084 22543 raft_consensus.cc:513] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.076404 22538 consensus_queue.cc:260] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.081121 22545 leader_election.cc:290] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979), 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:26.085316 22543 leader_election.cc:290] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979), 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015)
I20250109 21:00:26.110524 22539 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: No bootstrap required, opened a new log
I20250109 21:00:26.111135 22539 ts_tablet_manager.cc:1397] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent bootstrapping tablet: real 0.070s user 0.043s sys 0.025s
I20250109 21:00:26.112843 22538 ts_tablet_manager.cc:1428] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: Time spent starting tablet: real 0.041s user 0.007s sys 0.007s
I20250109 21:00:26.113925 22539 raft_consensus.cc:357] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.114955 22539 raft_consensus.cc:383] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:26.117096 22539 raft_consensus.cc:738] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d140c631d74fe99a0d50ffd8a7fad1, State: Initialized, Role: FOLLOWER
I20250109 21:00:26.117837 22539 consensus_queue.cc:260] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.119710 22343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f400fe6396994714a222603d757ed895" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" is_pre_election: true
I20250109 21:00:26.120258 22539 ts_tablet_manager.cc:1428] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250109 21:00:26.120525 22343 raft_consensus.cc:2463] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 0.
I20250109 21:00:26.121902 22381 leader_election.cc:304] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 06d140c631d74fe99a0d50ffd8a7fad1; no voters:
I20250109 21:00:26.122983 22556 raft_consensus.cc:2798] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:26.123435 22556 raft_consensus.cc:491] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:26.123811 22556 raft_consensus.cc:3054] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:26.126848 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f400fe6396994714a222603d757ed895" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97" is_pre_election: true
I20250109 21:00:26.127530 22492 raft_consensus.cc:2463] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 0.
I20250109 21:00:26.129177 22545 raft_consensus.cc:491] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:26.129626 22545 raft_consensus.cc:513] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.131192 22556 raft_consensus.cc:513] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.131517 22545 leader_election.cc:290] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979), 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:26.132056 22343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9aa6c74de24f019423b53c76147569" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" is_pre_election: true
I20250109 21:00:26.132586 22343 raft_consensus.cc:2463] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 0.
I20250109 21:00:26.132889 22556 leader_election.cc:290] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 1 election: Requested vote from peers 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979), 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:26.133808 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9aa6c74de24f019423b53c76147569" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97" is_pre_election: true
I20250109 21:00:26.134397 22492 raft_consensus.cc:2463] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 0.
I20250109 21:00:26.134738 22381 leader_election.cc:304] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 06d140c631d74fe99a0d50ffd8a7fad1; no voters:
I20250109 21:00:26.135418 22491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f400fe6396994714a222603d757ed895" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97"
I20250109 21:00:26.135948 22343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f400fe6396994714a222603d757ed895" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d140c631d74fe99a0d50ffd8a7fad1"
I20250109 21:00:26.136082 22491 raft_consensus.cc:3054] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:26.136365 22343 raft_consensus.cc:3054] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:26.140697 22343 raft_consensus.cc:2463] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 1.
I20250109 21:00:26.141434 22381 leader_election.cc:304] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 06d140c631d74fe99a0d50ffd8a7fad1; no voters:
I20250109 21:00:26.142305 22545 raft_consensus.cc:2798] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:26.143311 22343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9aa6c74de24f019423b53c76147569" candidate_uuid: "6a8775de2707400c87d2357cb2c09e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" is_pre_election: true
I20250109 21:00:26.143376 22491 raft_consensus.cc:2463] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 1.
I20250109 21:00:26.135435 22556 raft_consensus.cc:2798] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:26.145359 22556 raft_consensus.cc:491] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:26.145833 22556 raft_consensus.cc:3054] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:26.145892 22545 raft_consensus.cc:695] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 LEADER]: Becoming Leader. State: Replica: 02ad93d307eb412b9489918537829a87, State: Running, Role: LEADER
I20250109 21:00:26.144064 22343 raft_consensus.cc:2463] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a8775de2707400c87d2357cb2c09e97 in term 0.
I20250109 21:00:26.146929 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9aa6c74de24f019423b53c76147569" candidate_uuid: "6a8775de2707400c87d2357cb2c09e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ad93d307eb412b9489918537829a87" is_pre_election: true
I20250109 21:00:26.147519 22455 leader_election.cc:304] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d140c631d74fe99a0d50ffd8a7fad1, 6a8775de2707400c87d2357cb2c09e97; no voters:
I20250109 21:00:26.148173 22543 raft_consensus.cc:2798] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:26.148510 22543 raft_consensus.cc:491] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:26.148080 22545 consensus_queue.cc:237] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.148761 22543 raft_consensus.cc:3054] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:26.152295 22556 raft_consensus.cc:513] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.154079 22418 raft_consensus.cc:2388] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a8775de2707400c87d2357cb2c09e97 in current term 1: Already voted for candidate 02ad93d307eb412b9489918537829a87 in this term.
I20250109 21:00:26.154711 22556 leader_election.cc:290] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 1 election: Requested vote from peers 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979), 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:26.155107 22343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9aa6c74de24f019423b53c76147569" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d140c631d74fe99a0d50ffd8a7fad1"
I20250109 21:00:26.155262 22491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9aa6c74de24f019423b53c76147569" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97"
I20250109 21:00:26.155829 22343 raft_consensus.cc:3054] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:26.154247 22543 raft_consensus.cc:513] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.156705 22491 raft_consensus.cc:2388] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 02ad93d307eb412b9489918537829a87 in current term 1: Already voted for candidate 6a8775de2707400c87d2357cb2c09e97 in this term.
I20250109 21:00:26.158196 22239 catalog_manager.cc:5526] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02ad93d307eb412b9489918537829a87 (127.17.14.130). New cstate: current_term: 1 leader_uuid: "02ad93d307eb412b9489918537829a87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:26.159008 22543 leader_election.cc:290] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [CANDIDATE]: Term 1 election: Requested vote from peers 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979), 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015)
I20250109 21:00:26.160260 22342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9aa6c74de24f019423b53c76147569" candidate_uuid: "6a8775de2707400c87d2357cb2c09e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d140c631d74fe99a0d50ffd8a7fad1"
I20250109 21:00:26.160516 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9aa6c74de24f019423b53c76147569" candidate_uuid: "6a8775de2707400c87d2357cb2c09e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ad93d307eb412b9489918537829a87"
I20250109 21:00:26.161286 22418 raft_consensus.cc:2388] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6a8775de2707400c87d2357cb2c09e97 in current term 1: Already voted for candidate 02ad93d307eb412b9489918537829a87 in this term.
I20250109 21:00:26.162180 22343 raft_consensus.cc:2463] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 1.
I20250109 21:00:26.162181 22455 leader_election.cc:304] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a8775de2707400c87d2357cb2c09e97; no voters: 02ad93d307eb412b9489918537829a87, 06d140c631d74fe99a0d50ffd8a7fad1
I20250109 21:00:26.163324 22543 raft_consensus.cc:2743] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250109 21:00:26.163467 22381 leader_election.cc:304] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 06d140c631d74fe99a0d50ffd8a7fad1; no voters: 6a8775de2707400c87d2357cb2c09e97
I20250109 21:00:26.164222 22545 raft_consensus.cc:2798] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:26.164569 22545 raft_consensus.cc:695] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 LEADER]: Becoming Leader. State: Replica: 02ad93d307eb412b9489918537829a87, State: Running, Role: LEADER
I20250109 21:00:26.165287 22545 consensus_queue.cc:237] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.172410 22239 catalog_manager.cc:5526] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02ad93d307eb412b9489918537829a87 (127.17.14.130). New cstate: current_term: 1 leader_uuid: "02ad93d307eb412b9489918537829a87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:26.183109 22544 raft_consensus.cc:491] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:26.183486 22544 raft_consensus.cc:513] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.185237 22544 leader_election.cc:290] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015), 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:26.195286 22491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ff955074da4768beb84be0b6f54034" candidate_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97" is_pre_election: true
I20250109 21:00:26.195535 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ff955074da4768beb84be0b6f54034" candidate_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ad93d307eb412b9489918537829a87" is_pre_election: true
I20250109 21:00:26.195830 22491 raft_consensus.cc:2463] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06d140c631d74fe99a0d50ffd8a7fad1 in term 0.
I20250109 21:00:26.196075 22418 raft_consensus.cc:2463] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06d140c631d74fe99a0d50ffd8a7fad1 in term 0.
I20250109 21:00:26.196858 22309 leader_election.cc:304] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d140c631d74fe99a0d50ffd8a7fad1, 6a8775de2707400c87d2357cb2c09e97; no voters:
I20250109 21:00:26.197539 22544 raft_consensus.cc:2798] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:26.197804 22544 raft_consensus.cc:491] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:26.198037 22544 raft_consensus.cc:3054] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:26.199697 22543 raft_consensus.cc:491] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:26.200058 22543 raft_consensus.cc:513] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:26.201397 22543 leader_election.cc:290] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015), 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979)
I20250109 21:00:26.202134 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c3bf7218af4369bec2bdc85e3b5532" candidate_uuid: "6a8775de2707400c87d2357cb2c09e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ad93d307eb412b9489918537829a87" is_pre_election: true
I20250109 21:00:26.202577 22343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c3bf7218af4369bec2bdc85e3b5532" candidate_uuid: "6a8775de2707400c87d2357cb2c09e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" is_pre_election: true
I20250109 21:00:26.202723 22418 raft_consensus.cc:2463] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a8775de2707400c87d2357cb2c09e97 in term 0.
I20250109 21:00:26.203212 22343 raft_consensus.cc:2463] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a8775de2707400c87d2357cb2c09e97 in term 0.
I20250109 21:00:26.203066 22544 raft_consensus.cc:513] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.203841 22455 leader_election.cc:304] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 6a8775de2707400c87d2357cb2c09e97; no voters:
I20250109 21:00:26.204445 22543 raft_consensus.cc:2798] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:26.205137 22544 leader_election.cc:290] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [CANDIDATE]: Term 1 election: Requested vote from peers 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015), 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:26.205173 22543 raft_consensus.cc:491] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:26.205760 22543 raft_consensus.cc:3054] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:26.205758 22491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ff955074da4768beb84be0b6f54034" candidate_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97"
I20250109 21:00:26.205377 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ff955074da4768beb84be0b6f54034" candidate_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ad93d307eb412b9489918537829a87"
I20250109 21:00:26.206423 22491 raft_consensus.cc:3054] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:26.206815 22418 raft_consensus.cc:3054] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:26.211658 22418 raft_consensus.cc:2463] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06d140c631d74fe99a0d50ffd8a7fad1 in term 1.
I20250109 21:00:26.212172 22543 raft_consensus.cc:513] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:26.212579 22306 leader_election.cc:304] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 06d140c631d74fe99a0d50ffd8a7fad1; no voters:
I20250109 21:00:26.212622 22491 raft_consensus.cc:2463] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06d140c631d74fe99a0d50ffd8a7fad1 in term 1.
I20250109 21:00:26.213408 22544 raft_consensus.cc:2798] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:26.214357 22543 leader_election.cc:290] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [CANDIDATE]: Term 1 election: Requested vote from peers 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015), 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979)
I20250109 21:00:26.214875 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c3bf7218af4369bec2bdc85e3b5532" candidate_uuid: "6a8775de2707400c87d2357cb2c09e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ad93d307eb412b9489918537829a87"
I20250109 21:00:26.215260 22343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c3bf7218af4369bec2bdc85e3b5532" candidate_uuid: "6a8775de2707400c87d2357cb2c09e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d140c631d74fe99a0d50ffd8a7fad1"
I20250109 21:00:26.215610 22418 raft_consensus.cc:3054] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:26.215735 22343 raft_consensus.cc:3054] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:26.219951 22343 raft_consensus.cc:2463] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a8775de2707400c87d2357cb2c09e97 in term 1.
I20250109 21:00:26.220600 22544 raft_consensus.cc:695] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 LEADER]: Becoming Leader. State: Replica: 06d140c631d74fe99a0d50ffd8a7fad1, State: Running, Role: LEADER
I20250109 21:00:26.220927 22455 leader_election.cc:304] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d140c631d74fe99a0d50ffd8a7fad1, 6a8775de2707400c87d2357cb2c09e97; no voters:
I20250109 21:00:26.221813 22543 raft_consensus.cc:2798] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:26.221516 22544 consensus_queue.cc:237] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:26.222973 22418 raft_consensus.cc:2463] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a8775de2707400c87d2357cb2c09e97 in term 1.
I20250109 21:00:26.223670 22543 raft_consensus.cc:695] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 1 LEADER]: Becoming Leader. State: Replica: 6a8775de2707400c87d2357cb2c09e97, State: Running, Role: LEADER
I20250109 21:00:26.224378 22543 consensus_queue.cc:237] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:26.229540 22239 catalog_manager.cc:5526] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129). New cstate: current_term: 1 leader_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:26.231801 22238 catalog_manager.cc:5526] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131). New cstate: current_term: 1 leader_uuid: "6a8775de2707400c87d2357cb2c09e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:26.245790 17466 tablet_server.cc:178] TabletServer@127.17.14.129:0 shutting down...
I20250109 21:00:26.261689 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:26.262360 17466 tablet_replica.cc:331] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: stopping tablet replica
I20250109 21:00:26.262907 17466 raft_consensus.cc:2238] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:26.263439 17466 pending_rounds.cc:62] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: Trying to abort 1 pending ops.
I20250109 21:00:26.263633 17466 pending_rounds.cc:69] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112525521812807680 op_type: NO_OP noop_request { }
I20250109 21:00:26.263926 17466 raft_consensus.cc:2883] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 21:00:26.264179 17466 raft_consensus.cc:2267] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:26.266113 17466 tablet_replica.cc:331] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1: stopping tablet replica
I20250109 21:00:26.266577 17466 raft_consensus.cc:2238] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:26.266919 17466 raft_consensus.cc:2267] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:26.268473 17466 tablet_replica.cc:331] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: stopping tablet replica
I20250109 21:00:26.268903 17466 raft_consensus.cc:2238] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:26.269201 17466 raft_consensus.cc:2267] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:26.270686 17466 tablet_replica.cc:331] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: stopping tablet replica
I20250109 21:00:26.271092 17466 raft_consensus.cc:2238] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:26.271401 17466 raft_consensus.cc:2267] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:26.293123 17466 tablet_server.cc:195] TabletServer@127.17.14.129:0 shutdown complete.
I20250109 21:00:26.303314 17466 tablet_server.cc:178] TabletServer@127.17.14.130:0 shutting down...
I20250109 21:00:26.319358 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:26.320039 17466 tablet_replica.cc:331] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: stopping tablet replica
I20250109 21:00:26.320555 17466 raft_consensus.cc:2238] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:26.320930 17466 raft_consensus.cc:2267] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:26.322639 17466 tablet_replica.cc:331] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: stopping tablet replica
I20250109 21:00:26.323096 17466 raft_consensus.cc:2238] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:26.323598 17466 pending_rounds.cc:62] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: Trying to abort 1 pending ops.
I20250109 21:00:26.323773 17466 pending_rounds.cc:69] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112525521581080576 op_type: NO_OP noop_request { }
I20250109 21:00:26.324024 17466 raft_consensus.cc:2883] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 21:00:26.324258 17466 raft_consensus.cc:2267] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:26.326324 17466 tablet_replica.cc:331] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: stopping tablet replica
I20250109 21:00:26.326792 17466 raft_consensus.cc:2238] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:26.327275 17466 pending_rounds.cc:62] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: Trying to abort 1 pending ops.
I20250109 21:00:26.327440 17466 pending_rounds.cc:69] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112525521511776256 op_type: NO_OP noop_request { }
I20250109 21:00:26.327698 17466 raft_consensus.cc:2883] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 21:00:26.327929 17466 raft_consensus.cc:2267] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:26.329782 17466 tablet_replica.cc:331] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87: stopping tablet replica
I20250109 21:00:26.330209 17466 raft_consensus.cc:2238] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:26.330518 17466 raft_consensus.cc:2267] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:26.351644 17466 tablet_server.cc:195] TabletServer@127.17.14.130:0 shutdown complete.
I20250109 21:00:26.361550 17466 tablet_server.cc:178] TabletServer@127.17.14.131:0 shutting down...
I20250109 21:00:26.377077 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:26.377745 17466 tablet_replica.cc:331] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97: stopping tablet replica
I20250109 21:00:26.378319 17466 raft_consensus.cc:2238] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:26.378651 17466 raft_consensus.cc:2267] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:26.380234 17466 tablet_replica.cc:331] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97: stopping tablet replica
I20250109 21:00:26.380669 17466 raft_consensus.cc:2238] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:26.381016 17466 raft_consensus.cc:2267] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:26.382668 17466 tablet_replica.cc:331] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97: stopping tablet replica
I20250109 21:00:26.383042 17466 raft_consensus.cc:2238] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:26.383299 17466 raft_consensus.cc:2267] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:26.384752 17466 tablet_replica.cc:331] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: stopping tablet replica
I20250109 21:00:26.385160 17466 raft_consensus.cc:2238] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:26.385713 17466 pending_rounds.cc:62] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Trying to abort 1 pending ops.
I20250109 21:00:26.385862 17466 pending_rounds.cc:69] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112525521823911936 op_type: NO_OP noop_request { }
I20250109 21:00:26.386117 17466 raft_consensus.cc:2883] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 21:00:26.386349 17466 raft_consensus.cc:2267] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:26.397923 17466 tablet_server.cc:195] TabletServer@127.17.14.131:0 shutdown complete.
I20250109 21:00:26.410014 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:26.416483 22571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:26.417511 22570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:26.418684 17466 server_base.cc:1034] running on GCE node
W20250109 21:00:26.419402 22573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:26.420266 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:26.420568 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:26.420831 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456426420807 us; error 0 us; skew 500 ppm
I20250109 21:00:26.421329 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:26.428618 17466 webserver.cc:458] Webserver started at http://127.17.14.132:37955/ using document root <none> and password file <none>
I20250109 21:00:26.429150 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:26.429435 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:26.429816 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:26.430891 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-3-root/instance:
uuid: "b84cf1fa993c4b1ab4d020701560e45b"
format_stamp: "Formatted at 2025-01-09 21:00:26 on dist-test-slave-j54z"
I20250109 21:00:26.434935 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:00:26.437987 22578 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:26.438669 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:00:26.438992 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-3-root
uuid: "b84cf1fa993c4b1ab4d020701560e45b"
format_stamp: "Formatted at 2025-01-09 21:00:26 on dist-test-slave-j54z"
I20250109 21:00:26.439328 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:26.452913 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:26.454032 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:26.455336 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:26.457551 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:00:26.457849 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:26.458132 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:00:26.458376 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:26.492331 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.132:36117
I20250109 21:00:26.492415 22640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.132:36117 every 8 connection(s)
I20250109 21:00:26.505208 22641 heartbeater.cc:346] Connected to a master server at 127.17.14.190:39061
I20250109 21:00:26.505671 22641 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:26.506338 22641 heartbeater.cc:510] Master 127.17.14.190:39061 requested a full tablet report, sending...
I20250109 21:00:26.508167 22238 ts_manager.cc:194] Registered new tserver with Master: b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132:36117)
I20250109 21:00:26.509519 22238 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52514
W20250109 21:00:27.414098 22289 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:27.420282 22289 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:27.420713 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:27.424783 22289 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.14.129:44979: connect: Connection refused (error 111)
W20250109 21:00:27.425140 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.14.129:44979: connect: Connection refused (error 111)
W20250109 21:00:27.428812 22289 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.14.131:44163: connect: Connection refused (error 111)
W20250109 21:00:27.429148 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.14.131:44163: connect: Connection refused (error 111)
I20250109 21:00:27.511611 22641 heartbeater.cc:502] Master 127.17.14.190:39061 was elected leader, sending a full tablet report...
W20250109 21:00:28.458997 22289 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:28.464504 22289 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:28.464752 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:28.468652 22289 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.14.129:44979: connect: Connection refused (error 111)
W20250109 21:00:28.468869 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.14.129:44979: connect: Connection refused (error 111)
W20250109 21:00:28.473004 22289 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.14.131:44163: connect: Connection refused (error 111)
W20250109 21:00:28.473210 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.14.131:44163: connect: Connection refused (error 111)
W20250109 21:00:29.484336 22289 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.17.14.129:44979: connect: Connection refused (error 111)
W20250109 21:00:29.487090 22225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.14.129:44979: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20250109 21:00:29.489645 22289 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.14.129:44979: connect: Connection refused (error 111)
W20250109 21:00:29.489884 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.14.129:44979: connect: Connection refused (error 111)
W20250109 21:00:29.493842 22289 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.14.131:44163: connect: Connection refused (error 111)
W20250109 21:00:29.494081 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.14.131:44163: connect: Connection refused (error 111)
W20250109 21:00:29.497624 22289 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:29.497888 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:30.508613 22289 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:30.513780 22289 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:30.513998 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:30.517805 22289 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.14.131:44163: connect: Connection refused (error 111)
W20250109 21:00:30.518009 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.14.131:44163: connect: Connection refused (error 111)
W20250109 21:00:30.521447 22289 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:30.521701 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.14.130:39015: connect: Connection refused (error 111)
W20250109 21:00:31.523782 22289 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 6a8775de2707400c87d2357cb2c09e97 not available for placement
W20250109 21:00:31.852296 22637 debug-util.cc:398] Leaking SignalData structure 0x7b0800146040 after lost signal to thread 17469
W20250109 21:00:31.853204 22637 debug-util.cc:398] Leaking SignalData structure 0x7b0800212960 after lost signal to thread 22275
W20250109 21:00:31.854174 22637 debug-util.cc:398] Leaking SignalData structure 0x7b08002357e0 after lost signal to thread 22640
W20250109 21:00:32.525722 22289 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 6a8775de2707400c87d2357cb2c09e97 not available for placement
I20250109 21:00:33.062568 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:33.069204 22657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:33.072917 22658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:33.079983 22660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:33.081017 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:33.082410 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:33.082718 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:33.082934 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456433082917 us; error 0 us; skew 500 ppm
I20250109 21:00:33.083693 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:33.086779 17466 webserver.cc:458] Webserver started at http://127.17.14.129:44681/ using document root <none> and password file <none>
I20250109 21:00:33.087432 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:33.087663 17466 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250109 21:00:33.092633 17466 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250109 21:00:33.096601 22665 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:33.097625 17466 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:00:33.097977 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "06d140c631d74fe99a0d50ffd8a7fad1"
format_stamp: "Formatted at 2025-01-09 21:00:24 on dist-test-slave-j54z"
I20250109 21:00:33.098341 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:33.124186 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:33.125664 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:33.127482 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:33.132009 22672 ts_tablet_manager.cc:542] Loading tablet metadata (0/4 complete)
I20250109 21:00:33.151235 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250109 21:00:33.151633 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.021s user 0.002s sys 0.000s
I20250109 21:00:33.151952 17466 ts_tablet_manager.cc:594] Registering tablets (0/4 complete)
I20250109 21:00:33.159044 22672 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap starting.
I20250109 21:00:33.169590 22672 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:33.170289 22672 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap complete.
I20250109 21:00:33.170857 22672 ts_tablet_manager.cc:1397] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250109 21:00:33.172829 22672 raft_consensus.cc:357] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.173381 22672 raft_consensus.cc:738] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d140c631d74fe99a0d50ffd8a7fad1, State: Initialized, Role: FOLLOWER
I20250109 21:00:33.174078 22672 consensus_queue.cc:260] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.176777 17466 ts_tablet_manager.cc:610] Registered 4 tablets
I20250109 21:00:33.176788 22672 ts_tablet_manager.cc:1428] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250109 21:00:33.177266 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.025s user 0.025s sys 0.001s
I20250109 21:00:33.178002 22672 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap starting.
I20250109 21:00:33.193742 22672 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250109 21:00:33.195214 22672 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap complete.
I20250109 21:00:33.196192 22672 ts_tablet_manager.cc:1397] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.012s
I20250109 21:00:33.199707 22672 raft_consensus.cc:357] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.201072 22672 raft_consensus.cc:738] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d140c631d74fe99a0d50ffd8a7fad1, State: Initialized, Role: FOLLOWER
I20250109 21:00:33.201922 22672 consensus_queue.cc:260] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.204595 22672 ts_tablet_manager.cc:1428] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20250109 21:00:33.205535 22672 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap starting.
I20250109 21:00:33.220670 22672 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:33.221588 22672 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap complete.
I20250109 21:00:33.222345 22672 ts_tablet_manager.cc:1397] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.003s
I20250109 21:00:33.225205 22672 raft_consensus.cc:357] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.226052 22672 raft_consensus.cc:738] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d140c631d74fe99a0d50ffd8a7fad1, State: Initialized, Role: FOLLOWER
I20250109 21:00:33.226790 22672 consensus_queue.cc:260] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.229215 22672 ts_tablet_manager.cc:1428] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250109 21:00:33.229939 22672 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap starting.
I20250109 21:00:33.242214 22672 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:33.243088 22672 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: Bootstrap complete.
I20250109 21:00:33.243723 22672 ts_tablet_manager.cc:1397] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250109 21:00:33.244716 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:44979
I20250109 21:00:33.244863 22733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:44979 every 8 connection(s)
I20250109 21:00:33.245963 22672 raft_consensus.cc:357] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:33.246505 22672 raft_consensus.cc:738] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d140c631d74fe99a0d50ffd8a7fad1, State: Initialized, Role: FOLLOWER
I20250109 21:00:33.247110 22672 consensus_queue.cc:260] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:33.248977 22672 ts_tablet_manager.cc:1428] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 21:00:33.259259 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 21:00:33.275408 22734 heartbeater.cc:346] Connected to a master server at 127.17.14.190:39061
I20250109 21:00:33.275810 22734 heartbeater.cc:463] Registering TS with master...
W20250109 21:00:33.276170 22739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:33.276759 22734 heartbeater.cc:510] Master 127.17.14.190:39061 requested a full tablet report, sending...
W20250109 21:00:33.277980 22738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:33.281401 22238 ts_manager.cc:194] Re-registered known tserver with Master: 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979)
I20250109 21:00:33.296162 22238 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59812
I20250109 21:00:33.300101 22734 heartbeater.cc:502] Master 127.17.14.190:39061 was elected leader, sending a full tablet report...
W20250109 21:00:33.301075 22741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:33.302064 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:33.303076 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:33.303340 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:33.303517 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456433303499 us; error 0 us; skew 500 ppm
I20250109 21:00:33.304000 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:33.306907 17466 webserver.cc:458] Webserver started at http://127.17.14.130:37469/ using document root <none> and password file <none>
I20250109 21:00:33.307354 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:33.307525 17466 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250109 21:00:33.311524 17466 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20250109 21:00:33.315646 22746 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:33.316530 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:00:33.316883 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "02ad93d307eb412b9489918537829a87"
format_stamp: "Formatted at 2025-01-09 21:00:24 on dist-test-slave-j54z"
I20250109 21:00:33.317256 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:33.345214 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:33.346720 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:33.348621 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:33.371917 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250109 21:00:33.372275 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.002s sys 0.000s
I20250109 21:00:33.378890 22753 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: Bootstrap starting.
I20250109 21:00:33.389194 22753 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250109 21:00:33.389940 22753 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: Bootstrap complete.
I20250109 21:00:33.390434 22753 ts_tablet_manager.cc:1397] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250109 21:00:33.392140 22753 raft_consensus.cc:357] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.392786 22753 raft_consensus.cc:738] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ad93d307eb412b9489918537829a87, State: Initialized, Role: FOLLOWER
I20250109 21:00:33.393379 22753 consensus_queue.cc:260] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.395799 22753 ts_tablet_manager.cc:1428] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:00:33.396397 22753 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: Bootstrap starting.
I20250109 21:00:33.396649 17466 ts_tablet_manager.cc:610] Registered 4 tablets
I20250109 21:00:33.396962 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.024s user 0.022s sys 0.000s
I20250109 21:00:33.406531 22753 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:33.407308 22753 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: Bootstrap complete.
I20250109 21:00:33.407828 22753 ts_tablet_manager.cc:1397] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20250109 21:00:33.410169 22753 raft_consensus.cc:357] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.410933 22753 raft_consensus.cc:738] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ad93d307eb412b9489918537829a87, State: Initialized, Role: FOLLOWER
I20250109 21:00:33.411793 22753 consensus_queue.cc:260] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.414134 22753 ts_tablet_manager.cc:1428] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250109 21:00:33.414932 22753 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: Bootstrap starting.
I20250109 21:00:33.435914 22753 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250109 21:00:33.436939 22753 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: Bootstrap complete.
I20250109 21:00:33.437757 22753 ts_tablet_manager.cc:1397] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.012s
I20250109 21:00:33.440301 22753 raft_consensus.cc:357] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.441298 22753 raft_consensus.cc:738] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ad93d307eb412b9489918537829a87, State: Initialized, Role: FOLLOWER
I20250109 21:00:33.442097 22753 consensus_queue.cc:260] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.444011 22753 ts_tablet_manager.cc:1428] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:00:33.444604 22753 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87: Bootstrap starting.
I20250109 21:00:33.456619 22753 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:33.457542 22753 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87: Bootstrap complete.
I20250109 21:00:33.457935 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:39015
I20250109 21:00:33.458066 22814 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:39015 every 8 connection(s)
I20250109 21:00:33.458287 22753 ts_tablet_manager.cc:1397] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20250109 21:00:33.460098 22753 raft_consensus.cc:357] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:33.460598 22753 raft_consensus.cc:738] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ad93d307eb412b9489918537829a87, State: Initialized, Role: FOLLOWER
I20250109 21:00:33.461162 22753 consensus_queue.cc:260] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:33.464347 22753 ts_tablet_manager.cc:1428] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:00:33.464457 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:33.473335 22818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:33.475435 22819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:33.479525 17466 server_base.cc:1034] running on GCE node
W20250109 21:00:33.480720 22821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:33.481712 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:33.482034 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:33.482276 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456433482258 us; error 0 us; skew 500 ppm
I20250109 21:00:33.482949 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:33.484155 22815 heartbeater.cc:346] Connected to a master server at 127.17.14.190:39061
I20250109 21:00:33.484699 22815 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:33.485842 22815 heartbeater.cc:510] Master 127.17.14.190:39061 requested a full tablet report, sending...
I20250109 21:00:33.487586 17466 webserver.cc:458] Webserver started at http://127.17.14.131:46833/ using document root <none> and password file <none>
I20250109 21:00:33.488224 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:33.488415 17466 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250109 21:00:33.490309 22238 ts_manager.cc:194] Re-registered known tserver with Master: 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015)
I20250109 21:00:33.493527 17466 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20250109 21:00:33.496898 22238 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59824
I20250109 21:00:33.497808 22826 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:33.498767 17466 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:00:33.499032 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "6a8775de2707400c87d2357cb2c09e97"
format_stamp: "Formatted at 2025-01-09 21:00:24 on dist-test-slave-j54z"
I20250109 21:00:33.499320 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:33.499998 22815 heartbeater.cc:502] Master 127.17.14.190:39061 was elected leader, sending a full tablet report...
I20250109 21:00:33.512044 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:33.513144 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:33.514731 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20250109 21:00:33.528053 22289 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 6a8775de2707400c87d2357cb2c09e97 not available for placement
I20250109 21:00:33.540067 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250109 21:00:33.540439 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.023s user 0.000s sys 0.002s
I20250109 21:00:33.546917 22833 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap starting.
I20250109 21:00:33.559793 22833 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:33.560678 22833 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap complete.
I20250109 21:00:33.561369 22833 ts_tablet_manager.cc:1397] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20250109 21:00:33.563055 17466 ts_tablet_manager.cc:610] Registered 4 tablets
I20250109 21:00:33.563344 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.023s user 0.021s sys 0.000s
I20250109 21:00:33.563800 22833 raft_consensus.cc:357] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.564730 22833 raft_consensus.cc:738] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8775de2707400c87d2357cb2c09e97, State: Initialized, Role: FOLLOWER
I20250109 21:00:33.565528 22833 consensus_queue.cc:260] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.568560 22833 ts_tablet_manager.cc:1428] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250109 21:00:33.569370 22833 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap starting.
I20250109 21:00:33.585368 22833 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:33.586357 22833 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap complete.
I20250109 21:00:33.587126 22833 ts_tablet_manager.cc:1397] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20250109 21:00:33.590154 22833 raft_consensus.cc:357] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.591122 22833 raft_consensus.cc:738] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8775de2707400c87d2357cb2c09e97, State: Initialized, Role: FOLLOWER
I20250109 21:00:33.592197 22833 consensus_queue.cc:260] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.595584 22833 ts_tablet_manager.cc:1428] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20250109 21:00:33.596844 22833 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap starting.
I20250109 21:00:33.619369 22833 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:33.620568 22833 tablet_bootstrap.cc:492] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap complete.
I20250109 21:00:33.621521 22833 ts_tablet_manager.cc:1397] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20250109 21:00:33.624805 22833 raft_consensus.cc:357] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.625895 22833 raft_consensus.cc:738] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8775de2707400c87d2357cb2c09e97, State: Initialized, Role: FOLLOWER
I20250109 21:00:33.626928 22833 consensus_queue.cc:260] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:33.630054 22833 ts_tablet_manager.cc:1428] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250109 21:00:33.630970 22833 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap starting.
I20250109 21:00:33.635166 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:44163
I20250109 21:00:33.635291 22893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:44163 every 8 connection(s)
I20250109 21:00:33.654546 22833 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250109 21:00:33.655694 22833 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Bootstrap complete.
I20250109 21:00:33.656479 22833 ts_tablet_manager.cc:1397] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.013s
I20250109 21:00:33.658638 22833 raft_consensus.cc:357] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:33.659848 22833 raft_consensus.cc:738] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8775de2707400c87d2357cb2c09e97, State: Initialized, Role: FOLLOWER
I20250109 21:00:33.660735 22833 consensus_queue.cc:260] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:33.662750 22894 heartbeater.cc:346] Connected to a master server at 127.17.14.190:39061
I20250109 21:00:33.663252 22833 ts_tablet_manager.cc:1428] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Time spent starting tablet: real 0.006s user 0.002s sys 0.006s
I20250109 21:00:33.663288 22894 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:33.664750 22894 heartbeater.cc:510] Master 127.17.14.190:39061 requested a full tablet report, sending...
I20250109 21:00:33.674244 22238 ts_manager.cc:194] Re-registered known tserver with Master: 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:33.681826 22238 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59828
I20250109 21:00:33.685055 22894 heartbeater.cc:502] Master 127.17.14.190:39061 was elected leader, sending a full tablet report...
W20250109 21:00:34.560357 22289 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 02ad93d307eb412b9489918537829a87 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } }
W20250109 21:00:34.565438 22289 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f400fe6396994714a222603d757ed895, TS 06d140c631d74fe99a0d50ffd8a7fad1 -> TS b84cf1fa993c4b1ab4d020701560e45b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
W20250109 21:00:34.566033 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f400fe6396994714a222603d757ed895, TS 06d140c631d74fe99a0d50ffd8a7fad1 -> TS b84cf1fa993c4b1ab4d020701560e45b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
W20250109 21:00:34.579495 22289 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 76c3bf7218af4369bec2bdc85e3b5532, TS 02ad93d307eb412b9489918537829a87 -> TS b84cf1fa993c4b1ab4d020701560e45b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
W20250109 21:00:34.579912 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 76c3bf7218af4369bec2bdc85e3b5532, TS 02ad93d307eb412b9489918537829a87 -> TS b84cf1fa993c4b1ab4d020701560e45b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
W20250109 21:00:34.582966 22289 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7d9aa6c74de24f019423b53c76147569, TS 6a8775de2707400c87d2357cb2c09e97 -> TS b84cf1fa993c4b1ab4d020701560e45b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
W20250109 21:00:34.583328 22289 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7d9aa6c74de24f019423b53c76147569, TS 6a8775de2707400c87d2357cb2c09e97 -> TS b84cf1fa993c4b1ab4d020701560e45b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:34.604871 22902 raft_consensus.cc:491] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:34.605634 22902 raft_consensus.cc:513] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:34.607816 22902 leader_election.cc:290] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163), 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015)
I20250109 21:00:34.620265 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c3bf7218af4369bec2bdc85e3b5532" candidate_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97" is_pre_election: true
I20250109 21:00:34.620204 22789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c3bf7218af4369bec2bdc85e3b5532" candidate_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ad93d307eb412b9489918537829a87" is_pre_election: true
I20250109 21:00:34.621687 22868 raft_consensus.cc:2405] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06d140c631d74fe99a0d50ffd8a7fad1 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250109 21:00:34.621670 22789 raft_consensus.cc:2463] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06d140c631d74fe99a0d50ffd8a7fad1 in term 1.
I20250109 21:00:34.623085 22666 leader_election.cc:304] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 06d140c631d74fe99a0d50ffd8a7fad1; no voters: 6a8775de2707400c87d2357cb2c09e97
I20250109 21:00:34.623898 22902 raft_consensus.cc:2798] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250109 21:00:34.624296 22902 raft_consensus.cc:491] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:34.624758 22902 raft_consensus.cc:3054] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:34.631042 22902 raft_consensus.cc:513] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:34.632495 22902 leader_election.cc:290] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [CANDIDATE]: Term 2 election: Requested vote from peers 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163), 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015)
I20250109 21:00:34.633293 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c3bf7218af4369bec2bdc85e3b5532" candidate_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97"
I20250109 21:00:34.633395 22789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c3bf7218af4369bec2bdc85e3b5532" candidate_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ad93d307eb412b9489918537829a87"
I20250109 21:00:34.634052 22868 raft_consensus.cc:3054] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:34.634265 22789 raft_consensus.cc:3054] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:34.639987 22868 raft_consensus.cc:2405] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 06d140c631d74fe99a0d50ffd8a7fad1 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250109 21:00:34.640113 22789 raft_consensus.cc:2463] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06d140c631d74fe99a0d50ffd8a7fad1 in term 2.
I20250109 21:00:34.641273 22666 leader_election.cc:304] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 06d140c631d74fe99a0d50ffd8a7fad1; no voters: 6a8775de2707400c87d2357cb2c09e97
I20250109 21:00:34.642040 22902 raft_consensus.cc:2798] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 21:00:34.643137 22902 raft_consensus.cc:695] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 LEADER]: Becoming Leader. State: Replica: 06d140c631d74fe99a0d50ffd8a7fad1, State: Running, Role: LEADER
I20250109 21:00:34.644089 22902 consensus_queue.cc:237] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } }
I20250109 21:00:34.653066 22238 catalog_manager.cc:5526] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 reported cstate change: term changed from 1 to 2, leader changed from 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131) to 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129). New cstate: current_term: 2 leader_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: HEALTHY } } }
I20250109 21:00:34.653949 22907 raft_consensus.cc:491] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:34.654785 22907 raft_consensus.cc:513] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:34.657296 22907 leader_election.cc:290] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979), 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:34.673152 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f400fe6396994714a222603d757ed895" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97" is_pre_election: true
I20250109 21:00:34.673105 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f400fe6396994714a222603d757ed895" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" is_pre_election: true
I20250109 21:00:34.674279 22868 raft_consensus.cc:2463] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 1.
I20250109 21:00:34.674525 22708 raft_consensus.cc:2463] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 1.
I20250109 21:00:34.675654 22750 leader_election.cc:304] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 6a8775de2707400c87d2357cb2c09e97; no voters:
I20250109 21:00:34.676442 22907 raft_consensus.cc:2798] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250109 21:00:34.676846 22907 raft_consensus.cc:491] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:34.677201 22907 raft_consensus.cc:3054] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:34.682571 22907 raft_consensus.cc:513] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:34.684216 22907 leader_election.cc:290] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 2 election: Requested vote from peers 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979), 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:34.684991 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f400fe6396994714a222603d757ed895" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06d140c631d74fe99a0d50ffd8a7fad1"
I20250109 21:00:34.685285 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f400fe6396994714a222603d757ed895" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97"
I20250109 21:00:34.685745 22708 raft_consensus.cc:3054] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:34.685891 22868 raft_consensus.cc:3054] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:34.692874 22708 raft_consensus.cc:2463] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 2.
I20250109 21:00:34.692874 22868 raft_consensus.cc:2463] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 2.
I20250109 21:00:34.694193 22747 leader_election.cc:304] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 06d140c631d74fe99a0d50ffd8a7fad1; no voters:
I20250109 21:00:34.694833 22907 raft_consensus.cc:2798] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 21:00:34.696053 22907 raft_consensus.cc:695] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 2 LEADER]: Becoming Leader. State: Replica: 02ad93d307eb412b9489918537829a87, State: Running, Role: LEADER
I20250109 21:00:34.696939 22907 consensus_queue.cc:237] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:34.703228 22238 catalog_manager.cc:5526] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "02ad93d307eb412b9489918537829a87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:34.816308 22902 raft_consensus.cc:491] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:34.816820 22902 raft_consensus.cc:513] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:34.818285 22902 leader_election.cc:290] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015), 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:34.819159 22789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ff955074da4768beb84be0b6f54034" candidate_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02ad93d307eb412b9489918537829a87" is_pre_election: true
I20250109 21:00:34.819268 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ff955074da4768beb84be0b6f54034" candidate_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97" is_pre_election: true
I20250109 21:00:34.819725 22789 raft_consensus.cc:2463] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06d140c631d74fe99a0d50ffd8a7fad1 in term 1.
I20250109 21:00:34.819949 22868 raft_consensus.cc:2463] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06d140c631d74fe99a0d50ffd8a7fad1 in term 1.
I20250109 21:00:34.820624 22666 leader_election.cc:304] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 06d140c631d74fe99a0d50ffd8a7fad1; no voters:
I20250109 21:00:34.821358 22902 raft_consensus.cc:2798] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250109 21:00:34.821650 22902 raft_consensus.cc:491] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:34.821853 22902 raft_consensus.cc:3054] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:34.826380 22902 raft_consensus.cc:513] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:34.827692 22902 leader_election.cc:290] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [CANDIDATE]: Term 2 election: Requested vote from peers 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015), 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:34.827895 22907 raft_consensus.cc:491] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:34.828516 22789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ff955074da4768beb84be0b6f54034" candidate_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02ad93d307eb412b9489918537829a87"
I20250109 21:00:34.828385 22907 raft_consensus.cc:513] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:34.828733 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ff955074da4768beb84be0b6f54034" candidate_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97"
I20250109 21:00:34.829010 22789 raft_consensus.cc:3054] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:34.829283 22868 raft_consensus.cc:3054] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:34.831116 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9aa6c74de24f019423b53c76147569" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" is_pre_election: true
I20250109 21:00:34.831586 22867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9aa6c74de24f019423b53c76147569" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97" is_pre_election: true
I20250109 21:00:34.831869 22708 raft_consensus.cc:2463] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 1.
I20250109 21:00:34.832314 22867 raft_consensus.cc:2463] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 1.
I20250109 21:00:34.833178 22747 leader_election.cc:304] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 06d140c631d74fe99a0d50ffd8a7fad1; no voters:
I20250109 21:00:34.833642 22907 leader_election.cc:290] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979), 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:34.834342 22915 raft_consensus.cc:2798] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250109 21:00:34.834702 22915 raft_consensus.cc:491] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:34.835006 22915 raft_consensus.cc:3054] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:34.835713 22868 raft_consensus.cc:2463] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06d140c631d74fe99a0d50ffd8a7fad1 in term 2.
I20250109 21:00:34.835829 22789 raft_consensus.cc:2463] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06d140c631d74fe99a0d50ffd8a7fad1 in term 2.
I20250109 21:00:34.836740 22669 leader_election.cc:304] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d140c631d74fe99a0d50ffd8a7fad1, 6a8775de2707400c87d2357cb2c09e97; no voters:
I20250109 21:00:34.837563 22902 raft_consensus.cc:2798] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 21:00:34.838045 22902 raft_consensus.cc:695] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 LEADER]: Becoming Leader. State: Replica: 06d140c631d74fe99a0d50ffd8a7fad1, State: Running, Role: LEADER
I20250109 21:00:34.838937 22902 consensus_queue.cc:237] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:34.841588 22915 raft_consensus.cc:513] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:34.844079 22915 leader_election.cc:290] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 2 election: Requested vote from peers 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979), 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163)
I20250109 21:00:34.844301 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9aa6c74de24f019423b53c76147569" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06d140c631d74fe99a0d50ffd8a7fad1"
I20250109 21:00:34.844424 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9aa6c74de24f019423b53c76147569" candidate_uuid: "02ad93d307eb412b9489918537829a87" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a8775de2707400c87d2357cb2c09e97"
I20250109 21:00:34.844928 22708 raft_consensus.cc:3054] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:34.845031 22868 raft_consensus.cc:3054] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:34.846915 22238 catalog_manager.cc:5526] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:34.849414 22868 raft_consensus.cc:2463] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 2.
I20250109 21:00:34.850327 22750 leader_election.cc:304] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ad93d307eb412b9489918537829a87, 6a8775de2707400c87d2357cb2c09e97; no voters:
I20250109 21:00:34.851078 22915 raft_consensus.cc:2798] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 21:00:34.851593 22708 raft_consensus.cc:2463] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ad93d307eb412b9489918537829a87 in term 2.
I20250109 21:00:34.852283 22915 raft_consensus.cc:695] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 2 LEADER]: Becoming Leader. State: Replica: 02ad93d307eb412b9489918537829a87, State: Running, Role: LEADER
I20250109 21:00:34.853075 22915 consensus_queue.cc:237] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } }
I20250109 21:00:34.860054 22238 catalog_manager.cc:5526] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "02ad93d307eb412b9489918537829a87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:35.026046 22789 raft_consensus.cc:1270] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250109 21:00:35.027225 22902 consensus_queue.cc:1035] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:00:35.037192 22868 raft_consensus.cc:1270] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250109 21:00:35.037671 22868 pending_rounds.cc:77] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Aborting all ops after (but not including) 0
I20250109 21:00:35.037950 22868 pending_rounds.cc:99] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250109 21:00:35.038244 22868 raft_consensus.cc:2883] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250109 21:00:35.039865 22902 consensus_queue.cc:1035] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:00:35.217270 22708 raft_consensus.cc:1270] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Refusing update from remote peer 02ad93d307eb412b9489918537829a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 21:00:35.218578 22915 consensus_queue.cc:1035] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:00:35.229276 22868 raft_consensus.cc:1270] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Refusing update from remote peer 02ad93d307eb412b9489918537829a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 21:00:35.230916 22907 consensus_queue.cc:1035] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:00:35.250986 22708 raft_consensus.cc:1270] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Refusing update from remote peer 02ad93d307eb412b9489918537829a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 21:00:35.252054 22907 consensus_queue.cc:1035] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:00:35.263953 22868 raft_consensus.cc:1270] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Refusing update from remote peer 02ad93d307eb412b9489918537829a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 21:00:35.265103 22915 consensus_queue.cc:1035] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:00:35.301726 22789 raft_consensus.cc:1270] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 21:00:35.302857 22902 consensus_queue.cc:1035] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:00:35.314044 22868 raft_consensus.cc:1270] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 21:00:35.315528 22902 consensus_queue.cc:1035] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:00:35.600494 22708 consensus_queue.cc:237] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } }
I20250109 21:00:35.606709 22868 raft_consensus.cc:1270] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 21:00:35.607121 22789 raft_consensus.cc:1270] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 21:00:35.608381 22905 consensus_queue.cc:1035] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250109 21:00:35.609184 22932 consensus_queue.cc:1035] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 21:00:35.615141 22902 raft_consensus.cc:2949] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } } }
I20250109 21:00:35.616906 22789 raft_consensus.cc:2949] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } } }
I20250109 21:00:35.616773 22868 raft_consensus.cc:2949] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } } }
W20250109 21:00:35.618516 22666 consensus_peers.cc:487] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 -> Peer b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132:36117): Couldn't send request to peer b84cf1fa993c4b1ab4d020701560e45b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76c3bf7218af4369bec2bdc85e3b5532. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:35.627357 22708 consensus_queue.cc:237] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } }
I20250109 21:00:35.628933 22239 catalog_manager.cc:5526] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 reported cstate change: config changed from index -1 to 2, NON_VOTER b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) added. New cstate: current_term: 2 leader_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250109 21:00:35.635453 22666 consensus_peers.cc:487] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 -> Peer b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132:36117): Couldn't send request to peer b84cf1fa993c4b1ab4d020701560e45b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9ff955074da4768beb84be0b6f54034. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:35.635663 22868 raft_consensus.cc:1270] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 21:00:35.636243 22789 raft_consensus.cc:1270] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 21:00:35.637570 22902 consensus_queue.cc:1035] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:35.638324 22932 consensus_queue.cc:1035] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:35.646024 22905 raft_consensus.cc:2949] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } } }
I20250109 21:00:35.648711 22789 raft_consensus.cc:2949] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } } }
I20250109 21:00:35.649863 22868 raft_consensus.cc:2949] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } } }
I20250109 21:00:35.663443 22240 catalog_manager.cc:5526] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 reported cstate change: config changed from index -1 to 3, NON_VOTER b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) added. New cstate: current_term: 2 leader_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:35.665696 22789 consensus_queue.cc:237] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } }
I20250109 21:00:35.671916 22868 raft_consensus.cc:1270] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Refusing update from remote peer 02ad93d307eb412b9489918537829a87: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 21:00:35.673895 22915 consensus_queue.cc:1035] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:35.675174 22708 raft_consensus.cc:1270] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Refusing update from remote peer 02ad93d307eb412b9489918537829a87: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 21:00:35.681504 22915 consensus_queue.cc:1035] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:35.682999 22915 raft_consensus.cc:2949] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } } }
W20250109 21:00:35.686069 22747 consensus_peers.cc:487] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 -> Peer b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132:36117): Couldn't send request to peer b84cf1fa993c4b1ab4d020701560e45b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f400fe6396994714a222603d757ed895. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:35.685180 22867 raft_consensus.cc:2949] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } } }
I20250109 21:00:35.695427 22708 raft_consensus.cc:2949] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } } }
I20250109 21:00:35.696964 22239 catalog_manager.cc:5526] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 reported cstate change: config changed from index -1 to 3, NON_VOTER b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) added. New cstate: current_term: 2 leader_uuid: "02ad93d307eb412b9489918537829a87" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:36.019723 22942 ts_tablet_manager.cc:927] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b: Initiating tablet copy from peer 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979)
I20250109 21:00:36.021442 22942 tablet_copy_client.cc:323] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.129:44979
I20250109 21:00:36.032102 22718 tablet_copy_service.cc:140] P 06d140c631d74fe99a0d50ffd8a7fad1: Received BeginTabletCopySession request for tablet f9ff955074da4768beb84be0b6f54034 from peer b84cf1fa993c4b1ab4d020701560e45b ({username='slave'} at 127.0.0.1:33230)
I20250109 21:00:36.032723 22718 tablet_copy_service.cc:161] P 06d140c631d74fe99a0d50ffd8a7fad1: Beginning new tablet copy session on tablet f9ff955074da4768beb84be0b6f54034 from peer b84cf1fa993c4b1ab4d020701560e45b at {username='slave'} at 127.0.0.1:33230: session id = b84cf1fa993c4b1ab4d020701560e45b-f9ff955074da4768beb84be0b6f54034
I20250109 21:00:36.040227 22718 tablet_copy_source_session.cc:215] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 21:00:36.043378 22942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9ff955074da4768beb84be0b6f54034. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:36.057188 22942 tablet_copy_client.cc:806] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b: tablet copy: Starting download of 0 data blocks...
I20250109 21:00:36.057855 22942 tablet_copy_client.cc:670] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b: tablet copy: Starting download of 1 WAL segments...
I20250109 21:00:36.061285 22942 tablet_copy_client.cc:538] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 21:00:36.067770 22942 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b: Bootstrap starting.
I20250109 21:00:36.085996 22942 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:36.086874 22942 tablet_bootstrap.cc:492] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b: Bootstrap complete.
I20250109 21:00:36.087463 22942 ts_tablet_manager.cc:1397] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20250109 21:00:36.090157 22942 raft_consensus.cc:357] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } }
I20250109 21:00:36.090881 22942 raft_consensus.cc:738] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b84cf1fa993c4b1ab4d020701560e45b, State: Initialized, Role: LEARNER
I20250109 21:00:36.091449 22942 consensus_queue.cc:260] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } }
I20250109 21:00:36.095544 22942 ts_tablet_manager.cc:1428] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250109 21:00:36.097359 22718 tablet_copy_service.cc:342] P 06d140c631d74fe99a0d50ffd8a7fad1: Request end of tablet copy session b84cf1fa993c4b1ab4d020701560e45b-f9ff955074da4768beb84be0b6f54034 received from {username='slave'} at 127.0.0.1:33230
I20250109 21:00:36.097831 22718 tablet_copy_service.cc:434] P 06d140c631d74fe99a0d50ffd8a7fad1: ending tablet copy session b84cf1fa993c4b1ab4d020701560e45b-f9ff955074da4768beb84be0b6f54034 on tablet f9ff955074da4768beb84be0b6f54034 with peer b84cf1fa993c4b1ab4d020701560e45b
I20250109 21:00:36.149721 22942 ts_tablet_manager.cc:927] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b: Initiating tablet copy from peer 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015)
I20250109 21:00:36.151254 22942 tablet_copy_client.cc:323] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.130:39015
I20250109 21:00:36.159782 22799 tablet_copy_service.cc:140] P 02ad93d307eb412b9489918537829a87: Received BeginTabletCopySession request for tablet f400fe6396994714a222603d757ed895 from peer b84cf1fa993c4b1ab4d020701560e45b ({username='slave'} at 127.0.0.1:42614)
I20250109 21:00:36.160324 22799 tablet_copy_service.cc:161] P 02ad93d307eb412b9489918537829a87: Beginning new tablet copy session on tablet f400fe6396994714a222603d757ed895 from peer b84cf1fa993c4b1ab4d020701560e45b at {username='slave'} at 127.0.0.1:42614: session id = b84cf1fa993c4b1ab4d020701560e45b-f400fe6396994714a222603d757ed895
I20250109 21:00:36.167038 22799 tablet_copy_source_session.cc:215] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 21:00:36.169570 22942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f400fe6396994714a222603d757ed895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:36.180553 22942 tablet_copy_client.cc:806] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b: tablet copy: Starting download of 0 data blocks...
I20250109 21:00:36.181155 22942 tablet_copy_client.cc:670] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b: tablet copy: Starting download of 1 WAL segments...
I20250109 21:00:36.184439 22942 tablet_copy_client.cc:538] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 21:00:36.191056 22942 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b: Bootstrap starting.
I20250109 21:00:36.206800 22950 ts_tablet_manager.cc:927] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b: Initiating tablet copy from peer 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979)
I20250109 21:00:36.208622 22942 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:36.209022 22950 tablet_copy_client.cc:323] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.129:44979
I20250109 21:00:36.209391 22942 tablet_bootstrap.cc:492] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b: Bootstrap complete.
I20250109 21:00:36.210064 22942 ts_tablet_manager.cc:1397] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b: Time spent bootstrapping tablet: real 0.019s user 0.020s sys 0.000s
I20250109 21:00:36.211537 22718 tablet_copy_service.cc:140] P 06d140c631d74fe99a0d50ffd8a7fad1: Received BeginTabletCopySession request for tablet 76c3bf7218af4369bec2bdc85e3b5532 from peer b84cf1fa993c4b1ab4d020701560e45b ({username='slave'} at 127.0.0.1:33230)
I20250109 21:00:36.212034 22718 tablet_copy_service.cc:161] P 06d140c631d74fe99a0d50ffd8a7fad1: Beginning new tablet copy session on tablet 76c3bf7218af4369bec2bdc85e3b5532 from peer b84cf1fa993c4b1ab4d020701560e45b at {username='slave'} at 127.0.0.1:33230: session id = b84cf1fa993c4b1ab4d020701560e45b-76c3bf7218af4369bec2bdc85e3b5532
I20250109 21:00:36.212605 22942 raft_consensus.cc:357] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } }
I20250109 21:00:36.213486 22942 raft_consensus.cc:738] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b84cf1fa993c4b1ab4d020701560e45b, State: Initialized, Role: LEARNER
I20250109 21:00:36.214187 22942 consensus_queue.cc:260] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } }
I20250109 21:00:36.216578 22942 ts_tablet_manager.cc:1428] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20250109 21:00:36.218261 22799 tablet_copy_service.cc:342] P 02ad93d307eb412b9489918537829a87: Request end of tablet copy session b84cf1fa993c4b1ab4d020701560e45b-f400fe6396994714a222603d757ed895 received from {username='slave'} at 127.0.0.1:42614
I20250109 21:00:36.218667 22799 tablet_copy_service.cc:434] P 02ad93d307eb412b9489918537829a87: ending tablet copy session b84cf1fa993c4b1ab4d020701560e45b-f400fe6396994714a222603d757ed895 on tablet f400fe6396994714a222603d757ed895 with peer b84cf1fa993c4b1ab4d020701560e45b
I20250109 21:00:36.219652 22718 tablet_copy_source_session.cc:215] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 21:00:36.224284 22950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c3bf7218af4369bec2bdc85e3b5532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:36.235939 22950 tablet_copy_client.cc:806] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b: tablet copy: Starting download of 0 data blocks...
I20250109 21:00:36.236397 22950 tablet_copy_client.cc:670] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b: tablet copy: Starting download of 1 WAL segments...
I20250109 21:00:36.239333 22950 tablet_copy_client.cc:538] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 21:00:36.246348 22950 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b: Bootstrap starting.
I20250109 21:00:36.262929 22950 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:36.263618 22950 tablet_bootstrap.cc:492] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b: Bootstrap complete.
I20250109 21:00:36.264219 22950 ts_tablet_manager.cc:1397] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.001s
I20250109 21:00:36.266311 22950 raft_consensus.cc:357] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } }
I20250109 21:00:36.267071 22950 raft_consensus.cc:738] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b84cf1fa993c4b1ab4d020701560e45b, State: Initialized, Role: LEARNER
I20250109 21:00:36.267545 22950 consensus_queue.cc:260] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: true } }
I20250109 21:00:36.269728 22950 ts_tablet_manager.cc:1428] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20250109 21:00:36.271332 22718 tablet_copy_service.cc:342] P 06d140c631d74fe99a0d50ffd8a7fad1: Request end of tablet copy session b84cf1fa993c4b1ab4d020701560e45b-76c3bf7218af4369bec2bdc85e3b5532 received from {username='slave'} at 127.0.0.1:33230
I20250109 21:00:36.271742 22718 tablet_copy_service.cc:434] P 06d140c631d74fe99a0d50ffd8a7fad1: ending tablet copy session b84cf1fa993c4b1ab4d020701560e45b-76c3bf7218af4369bec2bdc85e3b5532 on tablet 76c3bf7218af4369bec2bdc85e3b5532 with peer b84cf1fa993c4b1ab4d020701560e45b
I20250109 21:00:36.592204 22616 raft_consensus.cc:1212] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250109 21:00:36.616389 22616 raft_consensus.cc:1212] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250109 21:00:36.731102 22616 raft_consensus.cc:1212] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250109 21:00:36.995644 22956 raft_consensus.cc:1059] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1: attempting to promote NON_VOTER b84cf1fa993c4b1ab4d020701560e45b to VOTER
I20250109 21:00:36.997537 22956 consensus_queue.cc:237] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } }
I20250109 21:00:37.003118 22616 raft_consensus.cc:1270] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b [term 2 LEARNER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:37.003221 22867 raft_consensus.cc:1270] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:37.004559 22789 raft_consensus.cc:1270] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:37.004513 22905 consensus_queue.cc:1035] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250109 21:00:37.005666 22956 consensus_queue.cc:1035] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:37.006512 22955 consensus_queue.cc:1035] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:37.015479 22955 raft_consensus.cc:2949] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.017138 22616 raft_consensus.cc:2949] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.020143 22867 raft_consensus.cc:2949] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.027364 22789 raft_consensus.cc:2949] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.028744 22240 catalog_manager.cc:5526] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b reported cstate change: config changed from index 3 to 4, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } attrs { replace: true } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.055943 22707 consensus_queue.cc:237] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } }
I20250109 21:00:37.060281 22867 raft_consensus.cc:1270] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 21:00:37.061501 22616 raft_consensus.cc:1270] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 21:00:37.061564 22905 consensus_queue.cc:1035] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 21:00:37.063254 22956 consensus_queue.cc:1035] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 21:00:37.068024 22955 raft_consensus.cc:2949] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 02ad93d307eb412b9489918537829a87 (127.17.14.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.069448 22867 raft_consensus.cc:2949] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 02ad93d307eb412b9489918537829a87 (127.17.14.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.071365 22616 raft_consensus.cc:2949] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 02ad93d307eb412b9489918537829a87 (127.17.14.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.076431 22283 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet f9ff955074da4768beb84be0b6f54034 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 21:00:37.079697 22239 catalog_manager.cc:5526] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 reported cstate change: config changed from index 4 to 5, VOTER 02ad93d307eb412b9489918537829a87 (127.17.14.130) evicted. New cstate: current_term: 2 leader_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250109 21:00:37.100258 22778 tablet_service.cc:1514] Processing DeleteTablet for tablet f9ff955074da4768beb84be0b6f54034 with delete_type TABLET_DATA_TOMBSTONED (TS 02ad93d307eb412b9489918537829a87 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42624
I20250109 21:00:37.102160 22968 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:37.102944 22968 raft_consensus.cc:2238] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:37.103470 22968 raft_consensus.cc:2267] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.106285 22968 ts_tablet_manager.cc:1905] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250109 21:00:37.119271 22968 ts_tablet_manager.cc:1918] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250109 21:00:37.119669 22968 log.cc:1198] T f9ff955074da4768beb84be0b6f54034 P 02ad93d307eb412b9489918537829a87: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-1-root/wals/f9ff955074da4768beb84be0b6f54034
I20250109 21:00:37.121109 22225 catalog_manager.cc:4872] TS 02ad93d307eb412b9489918537829a87 (127.17.14.130:39015): tablet f9ff955074da4768beb84be0b6f54034 (table test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]) successfully deleted
I20250109 21:00:37.209290 22954 raft_consensus.cc:1059] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87: attempting to promote NON_VOTER b84cf1fa993c4b1ab4d020701560e45b to VOTER
I20250109 21:00:37.210944 22954 consensus_queue.cc:237] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } }
I20250109 21:00:37.216481 22616 raft_consensus.cc:1270] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b [term 2 LEARNER]: Refusing update from remote peer 02ad93d307eb412b9489918537829a87: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:37.218271 22971 consensus_queue.cc:1035] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250109 21:00:37.219317 22867 raft_consensus.cc:1270] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Refusing update from remote peer 02ad93d307eb412b9489918537829a87: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:37.219974 22708 raft_consensus.cc:1270] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Refusing update from remote peer 02ad93d307eb412b9489918537829a87: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:37.220927 22971 consensus_queue.cc:1035] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:37.222848 22969 consensus_queue.cc:1035] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250109 21:00:37.233214 22969 raft_consensus.cc:2949] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.234647 22708 raft_consensus.cc:2949] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.238454 22955 raft_consensus.cc:1059] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1: attempting to promote NON_VOTER b84cf1fa993c4b1ab4d020701560e45b to VOTER
I20250109 21:00:37.238675 22867 raft_consensus.cc:2949] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.240895 22955 consensus_queue.cc:237] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } }
I20250109 21:00:37.246902 22241 catalog_manager.cc:5526] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 reported cstate change: config changed from index 3 to 4, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "02ad93d307eb412b9489918537829a87" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250109 21:00:37.242916 22616 raft_consensus.cc:2949] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.254729 22615 raft_consensus.cc:1270] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b [term 2 LEARNER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 21:00:37.258249 22789 raft_consensus.cc:1270] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 21:00:37.260250 22956 consensus_queue.cc:1035] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:37.261118 22867 raft_consensus.cc:1270] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 21:00:37.261739 22905 consensus_queue.cc:1035] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:37.263471 22905 consensus_queue.cc:1035] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:37.275475 22789 consensus_queue.cc:237] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } }
I20250109 21:00:37.281236 22867 raft_consensus.cc:1270] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Refusing update from remote peer 02ad93d307eb412b9489918537829a87: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 21:00:37.282146 22615 raft_consensus.cc:1270] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Refusing update from remote peer 02ad93d307eb412b9489918537829a87: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 21:00:37.280216 22905 raft_consensus.cc:2949] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.283567 22969 consensus_queue.cc:1035] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 21:00:37.285326 22969 consensus_queue.cc:1035] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 21:00:37.285065 22867 raft_consensus.cc:2949] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.290558 22616 raft_consensus.cc:2949] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.300422 22971 raft_consensus.cc:2949] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.303051 22239 catalog_manager.cc:5526] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 reported cstate change: config changed from index 2 to 3, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250109 21:00:37.305264 22868 raft_consensus.cc:2949] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.310415 22283 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet f400fe6396994714a222603d757ed895 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 21:00:37.310348 22787 raft_consensus.cc:2949] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, b84cf1fa993c4b1ab4d020701560e45b (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } attrs { replace: true } } peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.316843 22241 catalog_manager.cc:5526] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 reported cstate change: config changed from index 4 to 5, VOTER 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129) evicted. New cstate: current_term: 2 leader_uuid: "02ad93d307eb412b9489918537829a87" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:37.324563 22615 raft_consensus.cc:2949] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "6a8775de2707400c87d2357cb2c09e97" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44163 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.329922 22707 consensus_queue.cc:237] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } }
I20250109 21:00:37.342432 22787 raft_consensus.cc:1270] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:37.343824 22955 consensus_queue.cc:1035] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:37.342432 22615 raft_consensus.cc:1270] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Refusing update from remote peer 06d140c631d74fe99a0d50ffd8a7fad1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:37.347867 22955 consensus_queue.cc:1035] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:37.352618 22905 raft_consensus.cc:2949] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.354188 22787 raft_consensus.cc:2949] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.355486 22697 tablet_service.cc:1514] Processing DeleteTablet for tablet f400fe6396994714a222603d757ed895 with delete_type TABLET_DATA_TOMBSTONED (TS 06d140c631d74fe99a0d50ffd8a7fad1 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33242
I20250109 21:00:37.357668 22982 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:37.358562 22982 raft_consensus.cc:2238] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:37.358327 22615 raft_consensus.cc:2949] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } } }
I20250109 21:00:37.359390 22982 raft_consensus.cc:2267] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.362954 22982 ts_tablet_manager.cc:1905] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250109 21:00:37.364143 22283 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 76c3bf7218af4369bec2bdc85e3b5532 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 21:00:37.370621 22239 catalog_manager.cc:5526] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 reported cstate change: config changed from index 3 to 4, VOTER 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131) evicted. New cstate: current_term: 2 leader_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02ad93d307eb412b9489918537829a87" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06d140c631d74fe99a0d50ffd8a7fad1" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 44979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b84cf1fa993c4b1ab4d020701560e45b" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250109 21:00:37.377517 22982 ts_tablet_manager.cc:1918] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250109 21:00:37.377950 22982 log.cc:1198] T f400fe6396994714a222603d757ed895 P 06d140c631d74fe99a0d50ffd8a7fad1: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-0-root/wals/f400fe6396994714a222603d757ed895
I20250109 21:00:37.379215 22225 catalog_manager.cc:4872] TS 06d140c631d74fe99a0d50ffd8a7fad1 (127.17.14.129:44979): tablet f400fe6396994714a222603d757ed895 (table test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]) successfully deleted
I20250109 21:00:37.390519 22858 tablet_service.cc:1514] Processing DeleteTablet for tablet 76c3bf7218af4369bec2bdc85e3b5532 with delete_type TABLET_DATA_TOMBSTONED (TS 6a8775de2707400c87d2357cb2c09e97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50206
I20250109 21:00:37.392149 22984 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:37.392760 22984 raft_consensus.cc:2238] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:37.393254 22984 raft_consensus.cc:2267] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.395483 22984 ts_tablet_manager.cc:1905] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250109 21:00:37.405171 22984 ts_tablet_manager.cc:1918] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20250109 21:00:37.405445 22984 log.cc:1198] T 76c3bf7218af4369bec2bdc85e3b5532 P 6a8775de2707400c87d2357cb2c09e97: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736456270445890-17466-0/minicluster-data/ts-2-root/wals/76c3bf7218af4369bec2bdc85e3b5532
I20250109 21:00:37.406937 22228 catalog_manager.cc:4872] TS 6a8775de2707400c87d2357cb2c09e97 (127.17.14.131:44163): tablet 76c3bf7218af4369bec2bdc85e3b5532 (table test-workload [id=311bc3cddb274ce3babdd7dc87116c4d]) successfully deleted
I20250109 21:00:37.687970 17466 tablet_server.cc:178] TabletServer@127.17.14.129:44979 shutting down...
I20250109 21:00:37.707986 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:37.708792 17466 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:37.709291 17466 raft_consensus.cc:2238] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 LEADER]: Raft consensus shutting down.
I20250109 21:00:37.710091 17466 raft_consensus.cc:2267] T 76c3bf7218af4369bec2bdc85e3b5532 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.712319 17466 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:37.712793 17466 raft_consensus.cc:2238] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:37.713196 17466 raft_consensus.cc:2267] T 7d9aa6c74de24f019423b53c76147569 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.714865 17466 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:37.715284 17466 raft_consensus.cc:2238] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 LEADER]: Raft consensus shutting down.
I20250109 21:00:37.715971 17466 raft_consensus.cc:2267] T f9ff955074da4768beb84be0b6f54034 P 06d140c631d74fe99a0d50ffd8a7fad1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.739084 17466 tablet_server.cc:195] TabletServer@127.17.14.129:44979 shutdown complete.
I20250109 21:00:37.752712 17466 tablet_server.cc:178] TabletServer@127.17.14.130:39015 shutting down...
I20250109 21:00:37.773237 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:37.773939 17466 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:37.774545 17466 raft_consensus.cc:2238] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:37.775112 17466 raft_consensus.cc:2267] T 76c3bf7218af4369bec2bdc85e3b5532 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.777295 17466 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:37.777858 17466 raft_consensus.cc:2238] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 2 LEADER]: Raft consensus shutting down.
I20250109 21:00:37.778558 17466 raft_consensus.cc:2267] T 7d9aa6c74de24f019423b53c76147569 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.780252 17466 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:37.780659 17466 raft_consensus.cc:2238] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 2 LEADER]: Raft consensus shutting down.
I20250109 21:00:37.781291 17466 raft_consensus.cc:2267] T f400fe6396994714a222603d757ed895 P 02ad93d307eb412b9489918537829a87 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.803129 17466 tablet_server.cc:195] TabletServer@127.17.14.130:39015 shutdown complete.
I20250109 21:00:37.816020 17466 tablet_server.cc:178] TabletServer@127.17.14.131:44163 shutting down...
I20250109 21:00:37.835474 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:37.836267 17466 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:37.836854 17466 raft_consensus.cc:2238] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:37.837250 17466 raft_consensus.cc:2267] T 7d9aa6c74de24f019423b53c76147569 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.839072 17466 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:37.839489 17466 raft_consensus.cc:2238] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:37.839994 17466 raft_consensus.cc:2267] T f9ff955074da4768beb84be0b6f54034 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.842176 17466 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:37.842614 17466 raft_consensus.cc:2238] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:37.843114 17466 raft_consensus.cc:2267] T f400fe6396994714a222603d757ed895 P 6a8775de2707400c87d2357cb2c09e97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.863691 17466 tablet_server.cc:195] TabletServer@127.17.14.131:44163 shutdown complete.
I20250109 21:00:37.877254 17466 tablet_server.cc:178] TabletServer@127.17.14.132:0 shutting down...
I20250109 21:00:37.895968 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:37.896526 17466 tablet_replica.cc:331] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b: stopping tablet replica
I20250109 21:00:37.897064 17466 raft_consensus.cc:2238] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:37.897590 17466 raft_consensus.cc:2267] T 76c3bf7218af4369bec2bdc85e3b5532 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.899773 17466 tablet_replica.cc:331] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b: stopping tablet replica
I20250109 21:00:37.900216 17466 raft_consensus.cc:2238] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:37.900647 17466 raft_consensus.cc:2267] T f400fe6396994714a222603d757ed895 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.902518 17466 tablet_replica.cc:331] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b: stopping tablet replica
I20250109 21:00:37.902994 17466 raft_consensus.cc:2238] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:37.903422 17466 raft_consensus.cc:2267] T f9ff955074da4768beb84be0b6f54034 P b84cf1fa993c4b1ab4d020701560e45b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.935935 17466 tablet_server.cc:195] TabletServer@127.17.14.132:0 shutdown complete.
I20250109 21:00:37.947026 17466 master.cc:537] Master@127.17.14.190:39061 shutting down...
I20250109 21:00:37.963747 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:37.964217 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:37.964500 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P f6da49b7dc424a49aebed70a9e2d8c68: stopping tablet replica
I20250109 21:00:37.983796 17466 master.cc:559] Master@127.17.14.190:39061 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (13763 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20250109 21:00:38.019493 17466 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.14.190:38833
I20250109 21:00:38.020531 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:38.025575 22985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:38.026022 22986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:38.027057 17466 server_base.cc:1034] running on GCE node
W20250109 21:00:38.027456 22988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:38.028991 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:38.029194 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:38.029332 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456438029317 us; error 0 us; skew 500 ppm
I20250109 21:00:38.029812 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:38.037321 17466 webserver.cc:458] Webserver started at http://127.17.14.190:40533/ using document root <none> and password file <none>
I20250109 21:00:38.037814 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:38.037988 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:38.038244 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:38.039608 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/master-0-root/instance:
uuid: "bd6dfd4c014744338f9aa894a45e0eed"
format_stamp: "Formatted at 2025-01-09 21:00:38 on dist-test-slave-j54z"
I20250109 21:00:38.043938 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 21:00:38.047106 22993 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:38.047813 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:00:38.048071 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/master-0-root
uuid: "bd6dfd4c014744338f9aa894a45e0eed"
format_stamp: "Formatted at 2025-01-09 21:00:38 on dist-test-slave-j54z"
I20250109 21:00:38.048326 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:38.075860 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:38.077025 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:38.112568 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.190:38833
I20250109 21:00:38.112663 23044 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.190:38833 every 8 connection(s)
I20250109 21:00:38.116165 23045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:38.125716 23045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed: Bootstrap starting.
I20250109 21:00:38.129945 23045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:38.133842 23045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed: No bootstrap required, opened a new log
I20250109 21:00:38.135721 23045 raft_consensus.cc:357] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6dfd4c014744338f9aa894a45e0eed" member_type: VOTER }
I20250109 21:00:38.136111 23045 raft_consensus.cc:383] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:38.136340 23045 raft_consensus.cc:738] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6dfd4c014744338f9aa894a45e0eed, State: Initialized, Role: FOLLOWER
I20250109 21:00:38.136934 23045 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6dfd4c014744338f9aa894a45e0eed" member_type: VOTER }
I20250109 21:00:38.137359 23045 raft_consensus.cc:397] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:00:38.137588 23045 raft_consensus.cc:491] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:00:38.137835 23045 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:38.141969 23045 raft_consensus.cc:513] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6dfd4c014744338f9aa894a45e0eed" member_type: VOTER }
I20250109 21:00:38.142436 23045 leader_election.cc:304] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd6dfd4c014744338f9aa894a45e0eed; no voters:
I20250109 21:00:38.143666 23045 leader_election.cc:290] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:00:38.143997 23048 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:38.145417 23048 raft_consensus.cc:695] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [term 1 LEADER]: Becoming Leader. State: Replica: bd6dfd4c014744338f9aa894a45e0eed, State: Running, Role: LEADER
I20250109 21:00:38.146063 23048 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6dfd4c014744338f9aa894a45e0eed" member_type: VOTER }
I20250109 21:00:38.146667 23045 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:00:38.150507 23050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd6dfd4c014744338f9aa894a45e0eed. Latest consensus state: current_term: 1 leader_uuid: "bd6dfd4c014744338f9aa894a45e0eed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6dfd4c014744338f9aa894a45e0eed" member_type: VOTER } }
I20250109 21:00:38.150933 23050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [sys.catalog]: This master's current role is: LEADER
I20250109 21:00:38.152554 23049 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd6dfd4c014744338f9aa894a45e0eed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6dfd4c014744338f9aa894a45e0eed" member_type: VOTER } }
I20250109 21:00:38.153095 23049 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [sys.catalog]: This master's current role is: LEADER
I20250109 21:00:38.154888 23055 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:00:38.159680 23055 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:00:38.164057 17466 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250109 21:00:38.168253 23055 catalog_manager.cc:1348] Generated new cluster ID: dac5c2e835d94671af971c8bff8fe44b
I20250109 21:00:38.168716 23055 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:00:38.186671 23055 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:00:38.187937 23055 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:00:38.212605 23055 catalog_manager.cc:5899] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed: Generated new TSK 0
I20250109 21:00:38.213168 23055 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:00:38.231401 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:38.236693 23066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:38.237888 23067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:38.241389 23069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:38.242729 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:38.243491 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:38.243682 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:38.243829 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456438243812 us; error 0 us; skew 500 ppm
I20250109 21:00:38.244262 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:38.246505 17466 webserver.cc:458] Webserver started at http://127.17.14.129:42859/ using document root <none> and password file <none>
I20250109 21:00:38.246939 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:38.247114 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:38.247349 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:38.248472 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root/instance:
uuid: "06707233fdb442b79ca67eceabe36ebf"
format_stamp: "Formatted at 2025-01-09 21:00:38 on dist-test-slave-j54z"
I20250109 21:00:38.252799 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 21:00:38.255817 23074 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:38.256567 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:00:38.256799 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "06707233fdb442b79ca67eceabe36ebf"
format_stamp: "Formatted at 2025-01-09 21:00:38 on dist-test-slave-j54z"
I20250109 21:00:38.257040 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:38.272995 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:38.274092 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:38.275450 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:38.277609 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:00:38.277781 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:38.278033 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:00:38.278173 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:38.315213 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:46465
I20250109 21:00:38.315308 23136 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:46465 every 8 connection(s)
I20250109 21:00:38.319675 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:38.327481 23141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:38.327845 23142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:38.330426 23144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:38.331208 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:38.332444 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:38.332700 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:38.332746 23137 heartbeater.cc:346] Connected to a master server at 127.17.14.190:38833
I20250109 21:00:38.332901 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456438332886 us; error 0 us; skew 500 ppm
I20250109 21:00:38.333184 23137 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:38.333422 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:38.333940 23137 heartbeater.cc:510] Master 127.17.14.190:38833 requested a full tablet report, sending...
I20250109 21:00:38.336011 23010 ts_manager.cc:194] Registered new tserver with Master: 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465)
I20250109 21:00:38.336169 17466 webserver.cc:458] Webserver started at http://127.17.14.130:35511/ using document root <none> and password file <none>
I20250109 21:00:38.336694 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:38.336898 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:38.337141 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:38.337929 23010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40382
I20250109 21:00:38.338219 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-1-root/instance:
uuid: "b4a83a60513d47558a1b164c8631a921"
format_stamp: "Formatted at 2025-01-09 21:00:38 on dist-test-slave-j54z"
I20250109 21:00:38.342571 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250109 21:00:38.345726 23149 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:38.346390 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:00:38.346649 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-1-root
uuid: "b4a83a60513d47558a1b164c8631a921"
format_stamp: "Formatted at 2025-01-09 21:00:38 on dist-test-slave-j54z"
I20250109 21:00:38.346912 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:38.356889 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:38.357944 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:38.359364 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:38.361456 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:00:38.361699 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:38.361915 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:00:38.362063 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:38.398322 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.130:39647
I20250109 21:00:38.398414 23211 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.130:39647 every 8 connection(s)
I20250109 21:00:38.402887 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:38.409073 23215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:38.410600 23216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:38.414285 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:38.415050 23212 heartbeater.cc:346] Connected to a master server at 127.17.14.190:38833
I20250109 21:00:38.415501 23212 heartbeater.cc:463] Registering TS with master...
W20250109 21:00:38.415550 23219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:38.416471 23212 heartbeater.cc:510] Master 127.17.14.190:38833 requested a full tablet report, sending...
I20250109 21:00:38.416509 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:38.416878 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:38.417028 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456438417014 us; error 0 us; skew 500 ppm
I20250109 21:00:38.417737 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:38.418576 23010 ts_manager.cc:194] Registered new tserver with Master: b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647)
I20250109 21:00:38.420490 17466 webserver.cc:458] Webserver started at http://127.17.14.131:35893/ using document root <none> and password file <none>
I20250109 21:00:38.420495 23010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40394
I20250109 21:00:38.421233 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:38.421530 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:38.421864 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:38.422897 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-2-root/instance:
uuid: "6b2f500b0ae9460d8ea8defcb24c4939"
format_stamp: "Formatted at 2025-01-09 21:00:38 on dist-test-slave-j54z"
I20250109 21:00:38.427013 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20250109 21:00:38.429962 23223 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:38.430619 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20250109 21:00:38.430878 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-2-root
uuid: "6b2f500b0ae9460d8ea8defcb24c4939"
format_stamp: "Formatted at 2025-01-09 21:00:38 on dist-test-slave-j54z"
I20250109 21:00:38.431138 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:38.447772 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:38.448913 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:38.450357 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:38.452527 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:00:38.452725 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:38.453017 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:00:38.453215 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:38.492792 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.131:44721
I20250109 21:00:38.492887 23285 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.131:44721 every 8 connection(s)
I20250109 21:00:38.505204 23286 heartbeater.cc:346] Connected to a master server at 127.17.14.190:38833
I20250109 21:00:38.505585 23286 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:38.506287 23286 heartbeater.cc:510] Master 127.17.14.190:38833 requested a full tablet report, sending...
I20250109 21:00:38.508026 23010 ts_manager.cc:194] Registered new tserver with Master: 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:38.509093 17466 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013403079s
I20250109 21:00:38.509440 23010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40408
I20250109 21:00:39.340339 23137 heartbeater.cc:502] Master 127.17.14.190:38833 was elected leader, sending a full tablet report...
I20250109 21:00:39.423111 23212 heartbeater.cc:502] Master 127.17.14.190:38833 was elected leader, sending a full tablet report...
I20250109 21:00:39.511858 23286 heartbeater.cc:502] Master 127.17.14.190:38833 was elected leader, sending a full tablet report...
I20250109 21:00:39.541852 17466 test_util.cc:274] Using random seed: 1161769026
I20250109 21:00:39.563228 23010 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40412:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:00:39.565451 23010 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:00:39.621356 23251 tablet_service.cc:1467] Processing CreateTablet for tablet 883b7e05a84b48db811dbb6e7727b6bb (DEFAULT_TABLE table=test-workload [id=15e8613ceb444b6893261fdb1701be76]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 21:00:39.622350 23102 tablet_service.cc:1467] Processing CreateTablet for tablet 883b7e05a84b48db811dbb6e7727b6bb (DEFAULT_TABLE table=test-workload [id=15e8613ceb444b6893261fdb1701be76]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 21:00:39.622774 23251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883b7e05a84b48db811dbb6e7727b6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:39.622994 23175 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7958099f434d9baa13af9f2a891c4c (DEFAULT_TABLE table=test-workload [id=15e8613ceb444b6893261fdb1701be76]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 21:00:39.623603 23102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883b7e05a84b48db811dbb6e7727b6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:39.624285 23175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7958099f434d9baa13af9f2a891c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:39.623996 23249 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7958099f434d9baa13af9f2a891c4c (DEFAULT_TABLE table=test-workload [id=15e8613ceb444b6893261fdb1701be76]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 21:00:39.625196 23249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7958099f434d9baa13af9f2a891c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:39.626525 23101 tablet_service.cc:1467] Processing CreateTablet for tablet 65a5f4dded1343c08e21702f21074a86 (DEFAULT_TABLE table=test-workload [id=15e8613ceb444b6893261fdb1701be76]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250109 21:00:39.627082 23177 tablet_service.cc:1467] Processing CreateTablet for tablet 883b7e05a84b48db811dbb6e7727b6bb (DEFAULT_TABLE table=test-workload [id=15e8613ceb444b6893261fdb1701be76]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 21:00:39.627751 23101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65a5f4dded1343c08e21702f21074a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:39.628249 23177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883b7e05a84b48db811dbb6e7727b6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:39.630513 23100 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7958099f434d9baa13af9f2a891c4c (DEFAULT_TABLE table=test-workload [id=15e8613ceb444b6893261fdb1701be76]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 21:00:39.630633 23248 tablet_service.cc:1467] Processing CreateTablet for tablet 37d8b8c42dee49a49b83fdcf1cd89a9a (DEFAULT_TABLE table=test-workload [id=15e8613ceb444b6893261fdb1701be76]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 21:00:39.630895 23174 tablet_service.cc:1467] Processing CreateTablet for tablet 37d8b8c42dee49a49b83fdcf1cd89a9a (DEFAULT_TABLE table=test-workload [id=15e8613ceb444b6893261fdb1701be76]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 21:00:39.631724 23100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7958099f434d9baa13af9f2a891c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:39.631893 23248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d8b8c42dee49a49b83fdcf1cd89a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:39.632027 23174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d8b8c42dee49a49b83fdcf1cd89a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:39.622999 23250 tablet_service.cc:1467] Processing CreateTablet for tablet 65a5f4dded1343c08e21702f21074a86 (DEFAULT_TABLE table=test-workload [id=15e8613ceb444b6893261fdb1701be76]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250109 21:00:39.634382 23176 tablet_service.cc:1467] Processing CreateTablet for tablet 65a5f4dded1343c08e21702f21074a86 (DEFAULT_TABLE table=test-workload [id=15e8613ceb444b6893261fdb1701be76]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250109 21:00:39.635114 23099 tablet_service.cc:1467] Processing CreateTablet for tablet 37d8b8c42dee49a49b83fdcf1cd89a9a (DEFAULT_TABLE table=test-workload [id=15e8613ceb444b6893261fdb1701be76]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 21:00:39.635485 23250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65a5f4dded1343c08e21702f21074a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:39.636332 23099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d8b8c42dee49a49b83fdcf1cd89a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:39.635494 23176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65a5f4dded1343c08e21702f21074a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:39.645958 23306 tablet_bootstrap.cc:492] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939: Bootstrap starting.
I20250109 21:00:39.653430 23306 tablet_bootstrap.cc:654] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:39.668330 23308 tablet_bootstrap.cc:492] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921: Bootstrap starting.
I20250109 21:00:39.678097 23308 tablet_bootstrap.cc:654] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:39.692148 23308 tablet_bootstrap.cc:492] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921: No bootstrap required, opened a new log
I20250109 21:00:39.692693 23308 ts_tablet_manager.cc:1397] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.009s
I20250109 21:00:39.695290 23308 raft_consensus.cc:357] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.696250 23308 raft_consensus.cc:383] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:39.696589 23308 raft_consensus.cc:738] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a83a60513d47558a1b164c8631a921, State: Initialized, Role: FOLLOWER
I20250109 21:00:39.697428 23308 consensus_queue.cc:260] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.708142 23308 ts_tablet_manager.cc:1428] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921: Time spent starting tablet: real 0.015s user 0.011s sys 0.000s
I20250109 21:00:39.709118 23308 tablet_bootstrap.cc:492] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921: Bootstrap starting.
I20250109 21:00:39.714910 23308 tablet_bootstrap.cc:654] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:39.719835 23306 tablet_bootstrap.cc:492] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939: No bootstrap required, opened a new log
I20250109 21:00:39.720350 23306 ts_tablet_manager.cc:1397] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent bootstrapping tablet: real 0.075s user 0.035s sys 0.035s
I20250109 21:00:39.721609 23311 tablet_bootstrap.cc:492] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:39.723076 23306 raft_consensus.cc:357] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } }
I20250109 21:00:39.723913 23306 raft_consensus.cc:383] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:39.724215 23306 raft_consensus.cc:738] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2f500b0ae9460d8ea8defcb24c4939, State: Initialized, Role: FOLLOWER
I20250109 21:00:39.724838 23306 consensus_queue.cc:260] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } }
I20250109 21:00:39.725893 23308 tablet_bootstrap.cc:492] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921: No bootstrap required, opened a new log
I20250109 21:00:39.726402 23308 ts_tablet_manager.cc:1397] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.003s
I20250109 21:00:39.728647 23311 tablet_bootstrap.cc:654] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:39.729043 23308 raft_consensus.cc:357] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.729916 23308 raft_consensus.cc:383] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:39.730214 23306 ts_tablet_manager.cc:1428] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20250109 21:00:39.730234 23308 raft_consensus.cc:738] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a83a60513d47558a1b164c8631a921, State: Initialized, Role: FOLLOWER
I20250109 21:00:39.731266 23306 tablet_bootstrap.cc:492] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939: Bootstrap starting.
I20250109 21:00:39.731403 23308 consensus_queue.cc:260] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.736644 23308 ts_tablet_manager.cc:1428] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250109 21:00:39.737915 23308 tablet_bootstrap.cc:492] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921: Bootstrap starting.
I20250109 21:00:39.737979 23306 tablet_bootstrap.cc:654] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:39.742769 23311 tablet_bootstrap.cc:492] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf: No bootstrap required, opened a new log
I20250109 21:00:39.743306 23311 ts_tablet_manager.cc:1397] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.006s
I20250109 21:00:39.745762 23308 tablet_bootstrap.cc:654] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:39.745939 23311 raft_consensus.cc:357] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.746826 23311 raft_consensus.cc:383] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:39.747145 23311 raft_consensus.cc:738] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:39.747992 23311 consensus_queue.cc:260] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.752888 23306 tablet_bootstrap.cc:492] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939: No bootstrap required, opened a new log
I20250109 21:00:39.753203 23308 tablet_bootstrap.cc:492] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921: No bootstrap required, opened a new log
I20250109 21:00:39.753432 23306 ts_tablet_manager.cc:1397] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.000s
I20250109 21:00:39.753703 23308 ts_tablet_manager.cc:1397] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20250109 21:00:39.755065 23311 ts_tablet_manager.cc:1428] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.011s user 0.003s sys 0.005s
I20250109 21:00:39.756060 23311 tablet_bootstrap.cc:492] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:39.756274 23306 raft_consensus.cc:357] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.757099 23306 raft_consensus.cc:383] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:39.757426 23306 raft_consensus.cc:738] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2f500b0ae9460d8ea8defcb24c4939, State: Initialized, Role: FOLLOWER
I20250109 21:00:39.757534 23308 raft_consensus.cc:357] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } }
I20250109 21:00:39.758354 23308 raft_consensus.cc:383] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:39.758988 23308 raft_consensus.cc:738] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a83a60513d47558a1b164c8631a921, State: Initialized, Role: FOLLOWER
I20250109 21:00:39.758322 23306 consensus_queue.cc:260] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.759809 23308 consensus_queue.cc:260] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } }
I20250109 21:00:39.761456 23306 ts_tablet_manager.cc:1428] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250109 21:00:39.762223 23308 ts_tablet_manager.cc:1428] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250109 21:00:39.762528 23306 tablet_bootstrap.cc:492] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939: Bootstrap starting.
I20250109 21:00:39.763227 23308 tablet_bootstrap.cc:492] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921: Bootstrap starting.
I20250109 21:00:39.769050 23306 tablet_bootstrap.cc:654] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:39.769675 23308 tablet_bootstrap.cc:654] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:39.772341 23311 tablet_bootstrap.cc:654] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:39.780728 23311 tablet_bootstrap.cc:492] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: No bootstrap required, opened a new log
I20250109 21:00:39.781523 23306 tablet_bootstrap.cc:492] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939: No bootstrap required, opened a new log
I20250109 21:00:39.781688 23308 tablet_bootstrap.cc:492] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921: No bootstrap required, opened a new log
I20250109 21:00:39.781589 23311 ts_tablet_manager.cc:1397] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.010s
I20250109 21:00:39.782052 23306 ts_tablet_manager.cc:1397] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.006s
I20250109 21:00:39.782275 23308 ts_tablet_manager.cc:1397] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.012s
I20250109 21:00:39.784196 23306 raft_consensus.cc:357] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.784808 23306 raft_consensus.cc:383] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:39.784618 23311 raft_consensus.cc:357] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.784875 23308 raft_consensus.cc:357] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.785511 23311 raft_consensus.cc:383] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:39.785835 23308 raft_consensus.cc:383] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:39.785943 23311 raft_consensus.cc:738] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:39.786203 23308 raft_consensus.cc:738] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a83a60513d47558a1b164c8631a921, State: Initialized, Role: FOLLOWER
I20250109 21:00:39.785372 23306 raft_consensus.cc:738] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2f500b0ae9460d8ea8defcb24c4939, State: Initialized, Role: FOLLOWER
I20250109 21:00:39.786708 23311 consensus_queue.cc:260] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.786974 23308 consensus_queue.cc:260] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.787216 23306 consensus_queue.cc:260] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.789117 23311 ts_tablet_manager.cc:1428] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250109 21:00:39.789535 23308 ts_tablet_manager.cc:1428] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250109 21:00:39.789544 23306 ts_tablet_manager.cc:1428] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250109 21:00:39.790163 23311 tablet_bootstrap.cc:492] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:39.790194 23310 raft_consensus.cc:491] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:39.791090 23310 raft_consensus.cc:513] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.792117 23306 tablet_bootstrap.cc:492] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939: Bootstrap starting.
I20250109 21:00:39.814779 23311 tablet_bootstrap.cc:654] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:39.815783 23310 leader_election.cc:290] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:39.815997 23306 tablet_bootstrap.cc:654] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:39.824470 23112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a5f4dded1343c08e21702f21074a86" candidate_uuid: "b4a83a60513d47558a1b164c8631a921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf" is_pre_election: true
I20250109 21:00:39.825297 23112 raft_consensus.cc:2463] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a83a60513d47558a1b164c8631a921 in term 0.
I20250109 21:00:39.826591 23153 leader_election.cc:304] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, b4a83a60513d47558a1b164c8631a921; no voters:
I20250109 21:00:39.827162 23311 tablet_bootstrap.cc:492] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf: No bootstrap required, opened a new log
I20250109 21:00:39.827401 23306 tablet_bootstrap.cc:492] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939: No bootstrap required, opened a new log
I20250109 21:00:39.827463 23310 raft_consensus.cc:2798] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:39.827910 23306 ts_tablet_manager.cc:1397] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.015s
I20250109 21:00:39.827996 23310 raft_consensus.cc:491] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:39.828444 23310 raft_consensus.cc:3054] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:39.827623 23311 ts_tablet_manager.cc:1397] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.004s
I20250109 21:00:39.830350 23306 raft_consensus.cc:357] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.831131 23306 raft_consensus.cc:383] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:39.831458 23306 raft_consensus.cc:738] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2f500b0ae9460d8ea8defcb24c4939, State: Initialized, Role: FOLLOWER
I20250109 21:00:39.831913 23311 raft_consensus.cc:357] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } }
I20250109 21:00:39.832208 23306 consensus_queue.cc:260] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.832719 23311 raft_consensus.cc:383] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:39.833228 23311 raft_consensus.cc:738] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:39.835186 23306 ts_tablet_manager.cc:1428] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250109 21:00:39.834004 23311 consensus_queue.cc:260] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } }
I20250109 21:00:39.837088 23311 ts_tablet_manager.cc:1428] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.008s user 0.001s sys 0.004s
I20250109 21:00:39.837615 23261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a5f4dded1343c08e21702f21074a86" candidate_uuid: "b4a83a60513d47558a1b164c8631a921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" is_pre_election: true
I20250109 21:00:39.838094 23311 tablet_bootstrap.cc:492] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:39.838315 23261 raft_consensus.cc:2463] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a83a60513d47558a1b164c8631a921 in term 0.
I20250109 21:00:39.842365 23310 raft_consensus.cc:513] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.843892 23310 leader_election.cc:290] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [CANDIDATE]: Term 1 election: Requested vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:39.844455 23311 tablet_bootstrap.cc:654] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:39.844938 23112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a5f4dded1343c08e21702f21074a86" candidate_uuid: "b4a83a60513d47558a1b164c8631a921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf"
I20250109 21:00:39.845139 23261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a5f4dded1343c08e21702f21074a86" candidate_uuid: "b4a83a60513d47558a1b164c8631a921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2f500b0ae9460d8ea8defcb24c4939"
I20250109 21:00:39.845561 23112 raft_consensus.cc:3054] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:39.845697 23261 raft_consensus.cc:3054] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:39.852450 23112 raft_consensus.cc:2463] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a83a60513d47558a1b164c8631a921 in term 1.
I20250109 21:00:39.852450 23261 raft_consensus.cc:2463] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a83a60513d47558a1b164c8631a921 in term 1.
I20250109 21:00:39.853654 23153 leader_election.cc:304] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, b4a83a60513d47558a1b164c8631a921; no voters:
I20250109 21:00:39.854451 23310 raft_consensus.cc:2798] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:39.854811 23311 tablet_bootstrap.cc:492] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf: No bootstrap required, opened a new log
I20250109 21:00:39.855403 23311 ts_tablet_manager.cc:1397] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.001s
I20250109 21:00:39.855701 23310 raft_consensus.cc:695] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 1 LEADER]: Becoming Leader. State: Replica: b4a83a60513d47558a1b164c8631a921, State: Running, Role: LEADER
I20250109 21:00:39.856473 23310 consensus_queue.cc:237] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.857645 23311 raft_consensus.cc:357] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.858178 23311 raft_consensus.cc:383] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:39.858428 23311 raft_consensus.cc:738] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:39.858968 23311 consensus_queue.cc:260] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.861296 23311 ts_tablet_manager.cc:1428] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250109 21:00:39.864172 23007 catalog_manager.cc:5526] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4a83a60513d47558a1b164c8631a921 (127.17.14.130). New cstate: current_term: 1 leader_uuid: "b4a83a60513d47558a1b164c8631a921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:39.946336 23313 raft_consensus.cc:491] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:39.946866 23313 raft_consensus.cc:513] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } }
I20250109 21:00:39.949110 23313 leader_election.cc:290] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647)
I20250109 21:00:39.960285 23112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883b7e05a84b48db811dbb6e7727b6bb" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf" is_pre_election: true
I20250109 21:00:39.960537 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883b7e05a84b48db811dbb6e7727b6bb" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921" is_pre_election: true
I20250109 21:00:39.961081 23112 raft_consensus.cc:2463] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2f500b0ae9460d8ea8defcb24c4939 in term 0.
I20250109 21:00:39.961141 23187 raft_consensus.cc:2463] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2f500b0ae9460d8ea8defcb24c4939 in term 0.
I20250109 21:00:39.962224 23227 leader_election.cc:304] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, 6b2f500b0ae9460d8ea8defcb24c4939; no voters:
I20250109 21:00:39.962894 23313 raft_consensus.cc:2798] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:39.963213 23313 raft_consensus.cc:491] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:39.963462 23313 raft_consensus.cc:3054] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:39.967950 23313 raft_consensus.cc:513] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } }
I20250109 21:00:39.969173 23330 raft_consensus.cc:491] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:39.969446 23313 leader_election.cc:290] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 1 election: Requested vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647)
I20250109 21:00:39.969578 23330 raft_consensus.cc:513] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.970238 23112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883b7e05a84b48db811dbb6e7727b6bb" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf"
I20250109 21:00:39.970544 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883b7e05a84b48db811dbb6e7727b6bb" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921"
I20250109 21:00:39.970921 23112 raft_consensus.cc:3054] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:39.971055 23187 raft_consensus.cc:3054] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:39.971601 23330 leader_election.cc:290] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647)
I20250109 21:00:39.972342 23111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7958099f434d9baa13af9f2a891c4c" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf" is_pre_election: true
I20250109 21:00:39.972811 23111 raft_consensus.cc:2463] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2f500b0ae9460d8ea8defcb24c4939 in term 0.
I20250109 21:00:39.973693 23227 leader_election.cc:304] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, 6b2f500b0ae9460d8ea8defcb24c4939; no voters:
I20250109 21:00:39.974550 23330 raft_consensus.cc:2798] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:39.974607 23186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7958099f434d9baa13af9f2a891c4c" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921" is_pre_election: true
I20250109 21:00:39.975021 23330 raft_consensus.cc:491] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:39.975332 23186 raft_consensus.cc:2463] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2f500b0ae9460d8ea8defcb24c4939 in term 0.
I20250109 21:00:39.975790 23330 raft_consensus.cc:3054] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:39.977368 23112 raft_consensus.cc:2463] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2f500b0ae9460d8ea8defcb24c4939 in term 1.
I20250109 21:00:39.977366 23187 raft_consensus.cc:2463] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2f500b0ae9460d8ea8defcb24c4939 in term 1.
I20250109 21:00:39.978507 23227 leader_election.cc:304] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, 6b2f500b0ae9460d8ea8defcb24c4939; no voters:
I20250109 21:00:39.979089 23313 raft_consensus.cc:2798] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:39.980346 23313 raft_consensus.cc:695] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 LEADER]: Becoming Leader. State: Replica: 6b2f500b0ae9460d8ea8defcb24c4939, State: Running, Role: LEADER
I20250109 21:00:39.980830 23330 raft_consensus.cc:513] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:39.981117 23313 consensus_queue.cc:237] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } }
I20250109 21:00:39.982321 23330 leader_election.cc:290] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 1 election: Requested vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647)
I20250109 21:00:39.983304 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7958099f434d9baa13af9f2a891c4c" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921"
I20250109 21:00:39.983304 23112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7958099f434d9baa13af9f2a891c4c" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf"
I20250109 21:00:39.984090 23187 raft_consensus.cc:3054] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:39.984151 23112 raft_consensus.cc:3054] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:39.989820 23112 raft_consensus.cc:2463] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2f500b0ae9460d8ea8defcb24c4939 in term 1.
I20250109 21:00:39.989728 23008 catalog_manager.cc:5526] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131). New cstate: current_term: 1 leader_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:39.990754 23227 leader_election.cc:304] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, 6b2f500b0ae9460d8ea8defcb24c4939; no voters:
I20250109 21:00:39.990713 23187 raft_consensus.cc:2463] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2f500b0ae9460d8ea8defcb24c4939 in term 1.
I20250109 21:00:39.991626 23313 raft_consensus.cc:2798] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:39.992111 23313 raft_consensus.cc:695] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 LEADER]: Becoming Leader. State: Replica: 6b2f500b0ae9460d8ea8defcb24c4939, State: Running, Role: LEADER
I20250109 21:00:39.992790 23313 consensus_queue.cc:237] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.002008 23008 catalog_manager.cc:5526] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131). New cstate: current_term: 1 leader_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: HEALTHY } } }
I20250109 21:00:40.026823 23316 raft_consensus.cc:491] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:40.027354 23316 raft_consensus.cc:513] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.029430 23316 leader_election.cc:290] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647), 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:40.039188 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8b8c42dee49a49b83fdcf1cd89a9a" candidate_uuid: "06707233fdb442b79ca67eceabe36ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921" is_pre_election: true
I20250109 21:00:40.039391 23261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8b8c42dee49a49b83fdcf1cd89a9a" candidate_uuid: "06707233fdb442b79ca67eceabe36ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" is_pre_election: true
I20250109 21:00:40.039764 23187 raft_consensus.cc:2463] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06707233fdb442b79ca67eceabe36ebf in term 0.
I20250109 21:00:40.039970 23261 raft_consensus.cc:2463] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06707233fdb442b79ca67eceabe36ebf in term 0.
I20250109 21:00:40.040742 23075 leader_election.cc:304] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, b4a83a60513d47558a1b164c8631a921; no voters:
I20250109 21:00:40.041468 23316 raft_consensus.cc:2798] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:40.041777 23316 raft_consensus.cc:491] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:40.042021 23316 raft_consensus.cc:3054] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.046977 23316 raft_consensus.cc:513] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.048455 23316 leader_election.cc:290] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [CANDIDATE]: Term 1 election: Requested vote from peers b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647), 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:40.049273 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8b8c42dee49a49b83fdcf1cd89a9a" candidate_uuid: "06707233fdb442b79ca67eceabe36ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921"
I20250109 21:00:40.049350 23261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8b8c42dee49a49b83fdcf1cd89a9a" candidate_uuid: "06707233fdb442b79ca67eceabe36ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2f500b0ae9460d8ea8defcb24c4939"
I20250109 21:00:40.049929 23187 raft_consensus.cc:3054] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.050035 23261 raft_consensus.cc:3054] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.054291 23187 raft_consensus.cc:2463] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06707233fdb442b79ca67eceabe36ebf in term 1.
I20250109 21:00:40.054361 23261 raft_consensus.cc:2463] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06707233fdb442b79ca67eceabe36ebf in term 1.
I20250109 21:00:40.055199 23075 leader_election.cc:304] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, b4a83a60513d47558a1b164c8631a921; no voters:
I20250109 21:00:40.055827 23316 raft_consensus.cc:2798] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:40.057129 23316 raft_consensus.cc:695] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 1 LEADER]: Becoming Leader. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Running, Role: LEADER
I20250109 21:00:40.057837 23316 consensus_queue.cc:237] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.063715 23008 catalog_manager.cc:5526] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf reported cstate change: term changed from 0 to 1, leader changed from <none> to 06707233fdb442b79ca67eceabe36ebf (127.17.14.129). New cstate: current_term: 1 leader_uuid: "06707233fdb442b79ca67eceabe36ebf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:40.091857 17466 test_util.cc:274] Using random seed: 1162319033
I20250109 21:00:40.113063 23008 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40414:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:00:40.115108 23008 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:00:40.144580 23099 tablet_service.cc:1467] Processing CreateTablet for tablet 47c787e0ed42459da54d45a435f34bd9 (DEFAULT_TABLE table=dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 21:00:40.145298 23176 tablet_service.cc:1467] Processing CreateTablet for tablet 47c787e0ed42459da54d45a435f34bd9 (DEFAULT_TABLE table=dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 21:00:40.145889 23099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c787e0ed42459da54d45a435f34bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:40.146553 23176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c787e0ed42459da54d45a435f34bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:40.146342 23249 tablet_service.cc:1467] Processing CreateTablet for tablet 47c787e0ed42459da54d45a435f34bd9 (DEFAULT_TABLE table=dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250109 21:00:40.147475 23249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c787e0ed42459da54d45a435f34bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:40.149979 23177 tablet_service.cc:1467] Processing CreateTablet for tablet de00f78308924bc7aeec065d410f6c4f (DEFAULT_TABLE table=dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250109 21:00:40.151242 23177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de00f78308924bc7aeec065d410f6c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:40.159216 23175 tablet_service.cc:1467] Processing CreateTablet for tablet feec74a233bb410f893279c914f9128f (DEFAULT_TABLE table=dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 21:00:40.160388 23175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feec74a233bb410f893279c914f9128f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:40.160645 23248 tablet_service.cc:1467] Processing CreateTablet for tablet de00f78308924bc7aeec065d410f6c4f (DEFAULT_TABLE table=dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
W20250109 21:00:40.161115 23058 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250109 21:00:40.161469 23311 tablet_bootstrap.cc:492] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:40.161967 23248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de00f78308924bc7aeec065d410f6c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:40.164216 23174 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca95e9dff61489f98261bfd76c4cd3a (DEFAULT_TABLE table=dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 21:00:40.165369 23174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ca95e9dff61489f98261bfd76c4cd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:40.167409 23311 tablet_bootstrap.cc:654] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:40.169499 23102 tablet_service.cc:1467] Processing CreateTablet for tablet feec74a233bb410f893279c914f9128f (DEFAULT_TABLE table=dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 21:00:40.169911 23308 tablet_bootstrap.cc:492] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921: Bootstrap starting.
I20250109 21:00:40.170713 23102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feec74a233bb410f893279c914f9128f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:40.173821 23101 tablet_service.cc:1467] Processing CreateTablet for tablet de00f78308924bc7aeec065d410f6c4f (DEFAULT_TABLE table=dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250109 21:00:40.175007 23101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de00f78308924bc7aeec065d410f6c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:40.183599 23308 tablet_bootstrap.cc:654] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:40.186302 23250 tablet_service.cc:1467] Processing CreateTablet for tablet feec74a233bb410f893279c914f9128f (DEFAULT_TABLE table=dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250109 21:00:40.187979 23250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feec74a233bb410f893279c914f9128f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:40.199371 23099 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca95e9dff61489f98261bfd76c4cd3a (DEFAULT_TABLE table=dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 21:00:40.200331 23251 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca95e9dff61489f98261bfd76c4cd3a (DEFAULT_TABLE table=dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250109 21:00:40.201579 23251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ca95e9dff61489f98261bfd76c4cd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:40.202688 23308 tablet_bootstrap.cc:492] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921: No bootstrap required, opened a new log
I20250109 21:00:40.203152 23308 ts_tablet_manager.cc:1397] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.007s
I20250109 21:00:40.200681 23099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ca95e9dff61489f98261bfd76c4cd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:40.206300 23308 raft_consensus.cc:357] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.207024 23308 raft_consensus.cc:383] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:40.207262 23308 raft_consensus.cc:738] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a83a60513d47558a1b164c8631a921, State: Initialized, Role: FOLLOWER
I20250109 21:00:40.207952 23308 consensus_queue.cc:260] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.210430 23308 ts_tablet_manager.cc:1428] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250109 21:00:40.211374 23308 tablet_bootstrap.cc:492] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921: Bootstrap starting.
I20250109 21:00:40.218608 23308 tablet_bootstrap.cc:654] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:40.239984 23311 tablet_bootstrap.cc:492] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: No bootstrap required, opened a new log
I20250109 21:00:40.240521 23311 ts_tablet_manager.cc:1397] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.079s user 0.014s sys 0.052s
I20250109 21:00:40.243127 23311 raft_consensus.cc:357] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.243844 23311 raft_consensus.cc:383] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:40.244127 23311 raft_consensus.cc:738] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:40.244840 23311 consensus_queue.cc:260] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.250402 23311 ts_tablet_manager.cc:1428] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20250109 21:00:40.250909 23306 tablet_bootstrap.cc:492] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939: Bootstrap starting.
I20250109 21:00:40.251477 23311 tablet_bootstrap.cc:492] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:40.257217 23306 tablet_bootstrap.cc:654] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:40.260656 23308 tablet_bootstrap.cc:492] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921: No bootstrap required, opened a new log
I20250109 21:00:40.261032 23311 tablet_bootstrap.cc:654] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:40.261199 23308 ts_tablet_manager.cc:1397] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.007s
I20250109 21:00:40.264214 23310 consensus_queue.cc:1035] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250109 21:00:40.264171 23308 raft_consensus.cc:357] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.265612 23308 raft_consensus.cc:383] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:40.266064 23308 raft_consensus.cc:738] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a83a60513d47558a1b164c8631a921, State: Initialized, Role: FOLLOWER
I20250109 21:00:40.267026 23308 consensus_queue.cc:260] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.270215 23308 ts_tablet_manager.cc:1428] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20250109 21:00:40.272097 23308 tablet_bootstrap.cc:492] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921: Bootstrap starting.
I20250109 21:00:40.272984 23306 tablet_bootstrap.cc:492] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939: No bootstrap required, opened a new log
I20250109 21:00:40.273506 23306 ts_tablet_manager.cc:1397] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.012s
I20250109 21:00:40.276216 23306 raft_consensus.cc:357] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.276985 23306 raft_consensus.cc:383] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:40.277299 23306 raft_consensus.cc:738] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2f500b0ae9460d8ea8defcb24c4939, State: Initialized, Role: FOLLOWER
I20250109 21:00:40.278173 23306 consensus_queue.cc:260] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.280006 23311 tablet_bootstrap.cc:492] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: No bootstrap required, opened a new log
I20250109 21:00:40.280472 23311 ts_tablet_manager.cc:1397] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.006s
I20250109 21:00:40.281083 23306 ts_tablet_manager.cc:1428] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250109 21:00:40.282153 23306 tablet_bootstrap.cc:492] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939: Bootstrap starting.
I20250109 21:00:40.283236 23311 raft_consensus.cc:357] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.284672 23308 tablet_bootstrap.cc:654] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:40.289698 23306 tablet_bootstrap.cc:654] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:40.284082 23311 raft_consensus.cc:383] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:40.290450 23311 raft_consensus.cc:738] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:40.291249 23311 consensus_queue.cc:260] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.298800 23311 ts_tablet_manager.cc:1428] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.018s user 0.007s sys 0.002s
I20250109 21:00:40.299868 23311 tablet_bootstrap.cc:492] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:40.301795 23306 tablet_bootstrap.cc:492] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939: No bootstrap required, opened a new log
I20250109 21:00:40.302340 23306 ts_tablet_manager.cc:1397] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.012s
I20250109 21:00:40.305317 23306 raft_consensus.cc:357] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.306321 23306 raft_consensus.cc:383] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:40.306653 23306 raft_consensus.cc:738] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2f500b0ae9460d8ea8defcb24c4939, State: Initialized, Role: FOLLOWER
I20250109 21:00:40.307507 23308 tablet_bootstrap.cc:492] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921: No bootstrap required, opened a new log
I20250109 21:00:40.308087 23308 ts_tablet_manager.cc:1397] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.003s
I20250109 21:00:40.307466 23306 consensus_queue.cc:260] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.309027 23313 raft_consensus.cc:491] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:40.309442 23313 raft_consensus.cc:513] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.310609 23308 raft_consensus.cc:357] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.311269 23308 raft_consensus.cc:383] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:40.311565 23308 raft_consensus.cc:738] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a83a60513d47558a1b164c8631a921, State: Initialized, Role: FOLLOWER
I20250109 21:00:40.312232 23308 consensus_queue.cc:260] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.313791 23112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c787e0ed42459da54d45a435f34bd9" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf" is_pre_election: true
I20250109 21:00:40.314445 23112 raft_consensus.cc:2463] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2f500b0ae9460d8ea8defcb24c4939 in term 0.
I20250109 21:00:40.315699 23227 leader_election.cc:304] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, 6b2f500b0ae9460d8ea8defcb24c4939; no voters:
I20250109 21:00:40.316524 23330 raft_consensus.cc:2798] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:40.316692 23308 ts_tablet_manager.cc:1428] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20250109 21:00:40.316905 23330 raft_consensus.cc:491] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:40.317238 23330 raft_consensus.cc:3054] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.313295 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c787e0ed42459da54d45a435f34bd9" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921" is_pre_election: true
I20250109 21:00:40.318117 23187 raft_consensus.cc:2463] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2f500b0ae9460d8ea8defcb24c4939 in term 0.
I20250109 21:00:40.318280 23311 tablet_bootstrap.cc:654] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:40.321625 23313 leader_election.cc:290] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647)
I20250109 21:00:40.330278 23330 raft_consensus.cc:513] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.331771 23311 tablet_bootstrap.cc:492] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: No bootstrap required, opened a new log
I20250109 21:00:40.332371 23311 ts_tablet_manager.cc:1397] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.009s
I20250109 21:00:40.317691 23308 tablet_bootstrap.cc:492] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921: Bootstrap starting.
I20250109 21:00:40.335111 23112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c787e0ed42459da54d45a435f34bd9" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf"
I20250109 21:00:40.335765 23112 raft_consensus.cc:3054] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.336416 23306 ts_tablet_manager.cc:1428] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20250109 21:00:40.338199 23306 tablet_bootstrap.cc:492] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939: Bootstrap starting.
I20250109 21:00:40.343044 23308 tablet_bootstrap.cc:654] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:40.345584 23306 tablet_bootstrap.cc:654] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:40.336004 23311 raft_consensus.cc:357] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.346338 23311 raft_consensus.cc:383] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:40.346210 23326 consensus_queue.cc:1035] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250109 21:00:40.346683 23311 raft_consensus.cc:738] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:40.347404 23311 consensus_queue.cc:260] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.349964 23311 ts_tablet_manager.cc:1428] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.017s user 0.004s sys 0.003s
I20250109 21:00:40.350901 23311 tablet_bootstrap.cc:492] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:40.356868 23311 tablet_bootstrap.cc:654] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:40.372627 23311 tablet_bootstrap.cc:492] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: No bootstrap required, opened a new log
I20250109 21:00:40.373081 23311 ts_tablet_manager.cc:1397] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.003s
I20250109 21:00:40.375546 23311 raft_consensus.cc:357] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.376276 23311 raft_consensus.cc:383] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:40.376541 23311 raft_consensus.cc:738] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:40.376699 23112 raft_consensus.cc:2463] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2f500b0ae9460d8ea8defcb24c4939 in term 1.
I20250109 21:00:40.379083 23227 leader_election.cc:304] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, 6b2f500b0ae9460d8ea8defcb24c4939; no voters:
I20250109 21:00:40.380074 23313 raft_consensus.cc:2798] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:40.380685 23313 raft_consensus.cc:695] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 LEADER]: Becoming Leader. State: Replica: 6b2f500b0ae9460d8ea8defcb24c4939, State: Running, Role: LEADER
I20250109 21:00:40.381806 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c787e0ed42459da54d45a435f34bd9" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921"
I20250109 21:00:40.381464 23313 consensus_queue.cc:237] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.382472 23187 raft_consensus.cc:3054] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.390435 23187 raft_consensus.cc:2463] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2f500b0ae9460d8ea8defcb24c4939 in term 1.
I20250109 21:00:40.391705 23330 leader_election.cc:290] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 1 election: Requested vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647)
I20250109 21:00:40.396782 23311 consensus_queue.cc:260] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.398027 23010 catalog_manager.cc:5526] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131). New cstate: current_term: 1 leader_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: HEALTHY } } }
I20250109 21:00:40.399020 23311 ts_tablet_manager.cc:1428] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.026s user 0.004s sys 0.004s
I20250109 21:00:40.406874 23308 tablet_bootstrap.cc:492] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921: No bootstrap required, opened a new log
I20250109 21:00:40.407305 23308 ts_tablet_manager.cc:1397] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921: Time spent bootstrapping tablet: real 0.090s user 0.006s sys 0.022s
I20250109 21:00:40.410760 23306 tablet_bootstrap.cc:492] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939: No bootstrap required, opened a new log
I20250109 21:00:40.410805 23308 raft_consensus.cc:357] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.411715 23308 raft_consensus.cc:383] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:40.411998 23308 raft_consensus.cc:738] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a83a60513d47558a1b164c8631a921, State: Initialized, Role: FOLLOWER
I20250109 21:00:40.411401 23306 ts_tablet_manager.cc:1397] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent bootstrapping tablet: real 0.073s user 0.014s sys 0.036s
I20250109 21:00:40.413261 23313 consensus_queue.cc:1035] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:00:40.413170 23308 consensus_queue.cc:260] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.415372 23308 ts_tablet_manager.cc:1428] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250109 21:00:40.415829 23306 raft_consensus.cc:357] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.416543 23306 raft_consensus.cc:383] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:40.416810 23306 raft_consensus.cc:738] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2f500b0ae9460d8ea8defcb24c4939, State: Initialized, Role: FOLLOWER
I20250109 21:00:40.417595 23306 consensus_queue.cc:260] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.419656 23306 ts_tablet_manager.cc:1428] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20250109 21:00:40.422049 23306 tablet_bootstrap.cc:492] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939: Bootstrap starting.
I20250109 21:00:40.428081 23306 tablet_bootstrap.cc:654] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939: Neither blocks nor log segments found. Creating new log.
I20250109 21:00:40.428099 23331 consensus_queue.cc:1035] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250109 21:00:40.435022 23306 tablet_bootstrap.cc:492] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939: No bootstrap required, opened a new log
I20250109 21:00:40.435508 23306 ts_tablet_manager.cc:1397] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.001s
I20250109 21:00:40.437949 23306 raft_consensus.cc:357] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.438640 23306 raft_consensus.cc:383] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:00:40.438907 23306 raft_consensus.cc:738] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2f500b0ae9460d8ea8defcb24c4939, State: Initialized, Role: FOLLOWER
I20250109 21:00:40.439539 23306 consensus_queue.cc:260] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.441644 23306 ts_tablet_manager.cc:1428] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250109 21:00:40.443034 23316 raft_consensus.cc:491] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:40.443502 23316 raft_consensus.cc:513] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.445402 23316 leader_election.cc:290] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647), 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:40.445969 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feec74a233bb410f893279c914f9128f" candidate_uuid: "06707233fdb442b79ca67eceabe36ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921" is_pre_election: true
I20250109 21:00:40.446609 23187 raft_consensus.cc:2463] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06707233fdb442b79ca67eceabe36ebf in term 0.
I20250109 21:00:40.446981 23261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feec74a233bb410f893279c914f9128f" candidate_uuid: "06707233fdb442b79ca67eceabe36ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" is_pre_election: true
I20250109 21:00:40.447646 23075 leader_election.cc:304] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, b4a83a60513d47558a1b164c8631a921; no voters:
I20250109 21:00:40.447649 23261 raft_consensus.cc:2463] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06707233fdb442b79ca67eceabe36ebf in term 0.
I20250109 21:00:40.448630 23316 raft_consensus.cc:2798] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:40.448999 23316 raft_consensus.cc:491] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:40.449322 23316 raft_consensus.cc:3054] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.452947 23355 raft_consensus.cc:491] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:40.453334 23355 raft_consensus.cc:513] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.454782 23355 leader_election.cc:290] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:40.454511 23316 raft_consensus.cc:513] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.455637 23111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de00f78308924bc7aeec065d410f6c4f" candidate_uuid: "b4a83a60513d47558a1b164c8631a921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf" is_pre_election: true
I20250109 21:00:40.455766 23261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de00f78308924bc7aeec065d410f6c4f" candidate_uuid: "b4a83a60513d47558a1b164c8631a921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" is_pre_election: true
I20250109 21:00:40.456295 23111 raft_consensus.cc:2463] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a83a60513d47558a1b164c8631a921 in term 0.
I20250109 21:00:40.456367 23261 raft_consensus.cc:2463] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a83a60513d47558a1b164c8631a921 in term 0.
I20250109 21:00:40.457321 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feec74a233bb410f893279c914f9128f" candidate_uuid: "06707233fdb442b79ca67eceabe36ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921"
I20250109 21:00:40.457278 23153 leader_election.cc:304] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, b4a83a60513d47558a1b164c8631a921; no voters:
I20250109 21:00:40.457644 23260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feec74a233bb410f893279c914f9128f" candidate_uuid: "06707233fdb442b79ca67eceabe36ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2f500b0ae9460d8ea8defcb24c4939"
I20250109 21:00:40.458220 23260 raft_consensus.cc:3054] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.458307 23355 raft_consensus.cc:2798] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:40.460124 23355 raft_consensus.cc:491] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:40.460446 23355 raft_consensus.cc:3054] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.464049 23260 raft_consensus.cc:2463] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06707233fdb442b79ca67eceabe36ebf in term 1.
I20250109 21:00:40.458045 23187 raft_consensus.cc:3054] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.464479 23316 leader_election.cc:290] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [CANDIDATE]: Term 1 election: Requested vote from peers b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647), 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:40.464761 23355 raft_consensus.cc:513] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.465190 23078 leader_election.cc:304] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, 6b2f500b0ae9460d8ea8defcb24c4939; no voters:
I20250109 21:00:40.465997 23316 raft_consensus.cc:2798] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:40.466506 23355 leader_election.cc:290] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [CANDIDATE]: Term 1 election: Requested vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:40.467160 23316 raft_consensus.cc:695] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 1 LEADER]: Becoming Leader. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Running, Role: LEADER
I20250109 21:00:40.467990 23316 consensus_queue.cc:237] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.468644 23260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de00f78308924bc7aeec065d410f6c4f" candidate_uuid: "b4a83a60513d47558a1b164c8631a921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2f500b0ae9460d8ea8defcb24c4939"
I20250109 21:00:40.469231 23260 raft_consensus.cc:3054] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.470337 23187 raft_consensus.cc:2463] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06707233fdb442b79ca67eceabe36ebf in term 1.
I20250109 21:00:40.471369 23111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de00f78308924bc7aeec065d410f6c4f" candidate_uuid: "b4a83a60513d47558a1b164c8631a921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf"
I20250109 21:00:40.471976 23111 raft_consensus.cc:3054] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.476037 23260 raft_consensus.cc:2463] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a83a60513d47558a1b164c8631a921 in term 1.
I20250109 21:00:40.476017 23009 catalog_manager.cc:5526] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf reported cstate change: term changed from 0 to 1, leader changed from <none> to 06707233fdb442b79ca67eceabe36ebf (127.17.14.129). New cstate: current_term: 1 leader_uuid: "06707233fdb442b79ca67eceabe36ebf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:40.477196 23153 leader_election.cc:304] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b2f500b0ae9460d8ea8defcb24c4939, b4a83a60513d47558a1b164c8631a921; no voters:
I20250109 21:00:40.478040 23355 raft_consensus.cc:2798] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:40.478500 23355 raft_consensus.cc:695] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 1 LEADER]: Becoming Leader. State: Replica: b4a83a60513d47558a1b164c8631a921, State: Running, Role: LEADER
I20250109 21:00:40.478581 23111 raft_consensus.cc:2463] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a83a60513d47558a1b164c8631a921 in term 1.
I20250109 21:00:40.479324 23355 consensus_queue.cc:237] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.486569 23009 catalog_manager.cc:5526] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4a83a60513d47558a1b164c8631a921 (127.17.14.130). New cstate: current_term: 1 leader_uuid: "b4a83a60513d47558a1b164c8631a921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:40.532349 23316 raft_consensus.cc:491] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:00:40.532825 23316 raft_consensus.cc:513] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.534478 23316 leader_election.cc:290] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647), 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:40.535265 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca95e9dff61489f98261bfd76c4cd3a" candidate_uuid: "06707233fdb442b79ca67eceabe36ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921" is_pre_election: true
I20250109 21:00:40.535445 23260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca95e9dff61489f98261bfd76c4cd3a" candidate_uuid: "06707233fdb442b79ca67eceabe36ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" is_pre_election: true
I20250109 21:00:40.535832 23187 raft_consensus.cc:2463] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06707233fdb442b79ca67eceabe36ebf in term 0.
I20250109 21:00:40.536005 23260 raft_consensus.cc:2463] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06707233fdb442b79ca67eceabe36ebf in term 0.
I20250109 21:00:40.536784 23075 leader_election.cc:304] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, b4a83a60513d47558a1b164c8631a921; no voters:
I20250109 21:00:40.537381 23316 raft_consensus.cc:2798] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:00:40.537725 23316 raft_consensus.cc:491] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:00:40.538071 23316 raft_consensus.cc:3054] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.542847 23316 raft_consensus.cc:513] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.544307 23316 leader_election.cc:290] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [CANDIDATE]: Term 1 election: Requested vote from peers b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647), 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:40.544956 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca95e9dff61489f98261bfd76c4cd3a" candidate_uuid: "06707233fdb442b79ca67eceabe36ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921"
I20250109 21:00:40.545148 23260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca95e9dff61489f98261bfd76c4cd3a" candidate_uuid: "06707233fdb442b79ca67eceabe36ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2f500b0ae9460d8ea8defcb24c4939"
I20250109 21:00:40.545424 23187 raft_consensus.cc:3054] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.545631 23260 raft_consensus.cc:3054] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:00:40.550103 23187 raft_consensus.cc:2463] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06707233fdb442b79ca67eceabe36ebf in term 1.
I20250109 21:00:40.550117 23260 raft_consensus.cc:2463] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06707233fdb442b79ca67eceabe36ebf in term 1.
I20250109 21:00:40.551101 23075 leader_election.cc:304] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06707233fdb442b79ca67eceabe36ebf, b4a83a60513d47558a1b164c8631a921; no voters:
I20250109 21:00:40.551679 23316 raft_consensus.cc:2798] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:00:40.552089 23316 raft_consensus.cc:695] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 1 LEADER]: Becoming Leader. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Running, Role: LEADER
I20250109 21:00:40.552809 23316 consensus_queue.cc:237] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:40.559000 23009 catalog_manager.cc:5526] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf reported cstate change: term changed from 0 to 1, leader changed from <none> to 06707233fdb442b79ca67eceabe36ebf (127.17.14.129). New cstate: current_term: 1 leader_uuid: "06707233fdb442b79ca67eceabe36ebf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:40.591722 17466 tablet_server.cc:178] TabletServer@127.17.14.129:0 shutting down...
I20250109 21:00:40.594463 23331 consensus_queue.cc:1035] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 21:00:40.596583 23227 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20250109 21:00:40.601238 23227 consensus_peers.cc:487] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 -> Peer 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465): Couldn't send request to peer 06707233fdb442b79ca67eceabe36ebf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:40.604480 23316 consensus_queue.cc:1035] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250109 21:00:40.611960 23331 consensus_queue.cc:1035] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:00:40.618652 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:40.622633 23336 consensus_queue.cc:1035] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:00:40.624636 17466 tablet_replica.cc:331] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: stopping tablet replica
I20250109 21:00:40.625428 17466 raft_consensus.cc:2238] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:40.626166 17466 pending_rounds.cc:62] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Trying to abort 1 pending ops.
I20250109 21:00:40.626401 17466 pending_rounds.cc:69] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112525580510294016 op_type: NO_OP noop_request { }
I20250109 21:00:40.626819 17466 raft_consensus.cc:2883] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 21:00:40.627188 17466 raft_consensus.cc:2267] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:40.633909 17466 tablet_replica.cc:331] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: stopping tablet replica
I20250109 21:00:40.634649 17466 raft_consensus.cc:2238] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:40.635406 17466 pending_rounds.cc:62] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Trying to abort 1 pending ops.
I20250109 21:00:40.635648 17466 pending_rounds.cc:69] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112525580166098944 op_type: NO_OP noop_request { }
I20250109 21:00:40.636013 17466 raft_consensus.cc:2883] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 21:00:40.636335 17466 raft_consensus.cc:2267] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:40.639086 17466 tablet_replica.cc:331] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: stopping tablet replica
I20250109 21:00:40.639638 17466 raft_consensus.cc:2238] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:40.639967 17466 raft_consensus.cc:2267] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:40.641741 17466 tablet_replica.cc:331] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: stopping tablet replica
I20250109 21:00:40.642231 17466 raft_consensus.cc:2238] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:40.642544 17466 raft_consensus.cc:2267] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:40.644085 17466 tablet_replica.cc:331] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf: stopping tablet replica
I20250109 21:00:40.644537 17466 raft_consensus.cc:2238] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:40.645165 17466 raft_consensus.cc:2267] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:40.646878 17466 tablet_replica.cc:331] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf: stopping tablet replica
I20250109 21:00:40.647303 17466 raft_consensus.cc:2238] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:40.647656 17466 raft_consensus.cc:2267] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:40.649546 17466 tablet_replica.cc:331] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: stopping tablet replica
I20250109 21:00:40.650059 17466 raft_consensus.cc:2238] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:40.650409 17466 raft_consensus.cc:2267] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:40.651716 17466 tablet_replica.cc:331] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf: stopping tablet replica
I20250109 21:00:40.652076 17466 raft_consensus.cc:2238] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:40.652370 17466 raft_consensus.cc:2267] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:40.677805 17466 tablet_server.cc:195] TabletServer@127.17.14.129:0 shutdown complete.
W20250109 21:00:40.872139 23227 consensus_peers.cc:487] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 -> Peer 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465): Couldn't send request to peer 06707233fdb442b79ca67eceabe36ebf. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:46465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:40.876549 23355 consensus_queue.cc:1035] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 21:00:40.890607 23153 consensus_peers.cc:487] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 -> Peer 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465): Couldn't send request to peer 06707233fdb442b79ca67eceabe36ebf. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:46465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:40.922592 23313 consensus_queue.cc:1035] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 21:00:40.933692 23227 consensus_peers.cc:487] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 -> Peer 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465): Couldn't send request to peer 06707233fdb442b79ca67eceabe36ebf. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:46465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250109 21:00:40.951310 23153 consensus_peers.cc:487] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 -> Peer 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465): Couldn't send request to peer 06707233fdb442b79ca67eceabe36ebf. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:46465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:41.489559 23331 consensus_queue.cc:579] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Leader has been unable to successfully communicate with peer 06707233fdb442b79ca67eceabe36ebf for more than 1 seconds (1.056s)
I20250109 21:00:41.493469 23331 consensus_queue.cc:579] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Leader has been unable to successfully communicate with peer 06707233fdb442b79ca67eceabe36ebf for more than 1 seconds (1.110s)
I20250109 21:00:41.503422 23374 consensus_queue.cc:579] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [LEADER]: Leader has been unable to successfully communicate with peer 06707233fdb442b79ca67eceabe36ebf for more than 1 seconds (1.098s)
I20250109 21:00:41.704241 23010 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40412:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250109 21:00:41.704708 23010 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40412:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250109 21:00:41.718410 23010 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed: Sending DeleteTablet for 3 replicas of tablet 47c787e0ed42459da54d45a435f34bd9
I20250109 21:00:41.719614 23010 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed: Sending DeleteTablet for 3 replicas of tablet de00f78308924bc7aeec065d410f6c4f
I20250109 21:00:41.720397 23174 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c787e0ed42459da54d45a435f34bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:00:41 UTC) from {username='slave'} at 127.0.0.1:40616
I20250109 21:00:41.721081 23010 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed: Sending DeleteTablet for 3 replicas of tablet feec74a233bb410f893279c914f9128f
I20250109 21:00:41.721757 23251 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c787e0ed42459da54d45a435f34bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:00:41 UTC) from {username='slave'} at 127.0.0.1:47434
I20250109 21:00:41.722028 23250 tablet_service.cc:1514] Processing DeleteTablet for tablet de00f78308924bc7aeec065d410f6c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:00:41 UTC) from {username='slave'} at 127.0.0.1:47434
I20250109 21:00:41.722590 23010 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed: Sending DeleteTablet for 3 replicas of tablet 9ca95e9dff61489f98261bfd76c4cd3a
I20250109 21:00:41.723873 23388 tablet_replica.cc:331] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939: stopping tablet replica
I20250109 21:00:41.724576 23174 tablet_service.cc:1514] Processing DeleteTablet for tablet de00f78308924bc7aeec065d410f6c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:00:41 UTC) from {username='slave'} at 127.0.0.1:40616
I20250109 21:00:41.724857 23388 raft_consensus.cc:2238] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:41.725440 23175 tablet_service.cc:1514] Processing DeleteTablet for tablet feec74a233bb410f893279c914f9128f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:00:41 UTC) from {username='slave'} at 127.0.0.1:40616
I20250109 21:00:41.725931 23388 raft_consensus.cc:2267] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:41.723474 23387 tablet_replica.cc:331] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921: stopping tablet replica
I20250109 21:00:41.726876 23175 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca95e9dff61489f98261bfd76c4cd3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:00:41 UTC) from {username='slave'} at 127.0.0.1:40616
I20250109 21:00:41.727463 23387 raft_consensus.cc:2238] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:41.727777 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 21:00:41.728107 23387 raft_consensus.cc:2267] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:41.730042 23251 tablet_service.cc:1514] Processing DeleteTablet for tablet feec74a233bb410f893279c914f9128f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:00:41 UTC) from {username='slave'} at 127.0.0.1:47434
I20250109 21:00:41.730806 23388 ts_tablet_manager.cc:1905] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:00:41.730914 23387 ts_tablet_manager.cc:1905] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 21:00:41.731572 22997 catalog_manager.cc:4615] TS 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet feec74a233bb410f893279c914f9128f: Network error: Client connection negotiation failed: client connection to 127.17.14.129:46465: connect: Connection refused (error 111)
I20250109 21:00:41.731846 23251 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca95e9dff61489f98261bfd76c4cd3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:00:41 UTC) from {username='slave'} at 127.0.0.1:47434
W20250109 21:00:41.738235 23390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:41.743909 23391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:41.748939 23393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:41.749308 23388 ts_tablet_manager.cc:1918] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 21:00:41.749883 23388 log.cc:1198] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-2-root/wals/47c787e0ed42459da54d45a435f34bd9
I20250109 21:00:41.750022 17466 server_base.cc:1034] running on GCE node
I20250109 21:00:41.750613 23388 ts_tablet_manager.cc:1939] T 47c787e0ed42459da54d45a435f34bd9 P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting consensus metadata
I20250109 21:00:41.751164 23387 ts_tablet_manager.cc:1918] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 21:00:41.751451 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250109 21:00:41.751525 23387 log.cc:1198] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-1-root/wals/47c787e0ed42459da54d45a435f34bd9
W20250109 21:00:41.751861 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:41.752104 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456441752085 us; error 0 us; skew 500 ppm
I20250109 21:00:41.752385 23387 ts_tablet_manager.cc:1939] T 47c787e0ed42459da54d45a435f34bd9 P b4a83a60513d47558a1b164c8631a921: Deleting consensus metadata
I20250109 21:00:41.752873 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:41.753813 23388 tablet_replica.cc:331] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939: stopping tablet replica
I20250109 21:00:41.753885 22997 catalog_manager.cc:4872] TS 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721): tablet 47c787e0ed42459da54d45a435f34bd9 (table dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]) successfully deleted
I20250109 21:00:41.754773 23388 raft_consensus.cc:2238] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:41.755141 23331 consensus_queue.cc:579] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Leader has been unable to successfully communicate with peer 06707233fdb442b79ca67eceabe36ebf for more than 1 seconds (1.154s)
I20250109 21:00:41.755393 22994 catalog_manager.cc:4872] TS b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647): tablet 47c787e0ed42459da54d45a435f34bd9 (table dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]) successfully deleted
I20250109 21:00:41.755532 23388 raft_consensus.cc:2267] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:41.755141 23387 tablet_replica.cc:331] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921: stopping tablet replica
I20250109 21:00:41.757570 23387 raft_consensus.cc:2238] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:41.758649 23387 raft_consensus.cc:2267] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:41.759361 23388 ts_tablet_manager.cc:1905] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:00:41.761744 23387 ts_tablet_manager.cc:1905] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:00:41.761893 17466 webserver.cc:458] Webserver started at http://127.17.14.132:45539/ using document root <none> and password file <none>
I20250109 21:00:41.764847 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:41.765105 17466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:00:41.765522 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:00:41.766901 17466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-3-root/instance:
uuid: "19622fe17ad2450fb04e59fff92d9f5e"
format_stamp: "Formatted at 2025-01-09 21:00:41 on dist-test-slave-j54z"
I20250109 21:00:41.771588 17466 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250109 21:00:41.773522 23388 ts_tablet_manager.cc:1918] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 21:00:41.773907 23388 log.cc:1198] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-2-root/wals/de00f78308924bc7aeec065d410f6c4f
I20250109 21:00:41.774654 23388 ts_tablet_manager.cc:1939] T de00f78308924bc7aeec065d410f6c4f P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting consensus metadata
I20250109 21:00:41.774930 23398 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:41.775816 17466 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:00:41.775888 23387 ts_tablet_manager.cc:1918] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 21:00:41.776189 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-3-root
uuid: "19622fe17ad2450fb04e59fff92d9f5e"
format_stamp: "Formatted at 2025-01-09 21:00:41 on dist-test-slave-j54z"
I20250109 21:00:41.776391 23387 log.cc:1198] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-1-root/wals/de00f78308924bc7aeec065d410f6c4f
I20250109 21:00:41.776595 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:41.777169 23387 ts_tablet_manager.cc:1939] T de00f78308924bc7aeec065d410f6c4f P b4a83a60513d47558a1b164c8631a921: Deleting consensus metadata
I20250109 21:00:41.777887 23388 tablet_replica.cc:331] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939: stopping tablet replica
I20250109 21:00:41.778196 22997 catalog_manager.cc:4872] TS 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721): tablet de00f78308924bc7aeec065d410f6c4f (table dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]) successfully deleted
I20250109 21:00:41.778728 23388 raft_consensus.cc:2238] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:41.779263 23388 raft_consensus.cc:2267] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:41.779934 23387 tablet_replica.cc:331] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921: stopping tablet replica
I20250109 21:00:41.780066 22994 catalog_manager.cc:4872] TS b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647): tablet de00f78308924bc7aeec065d410f6c4f (table dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]) successfully deleted
I20250109 21:00:41.780797 23387 raft_consensus.cc:2238] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:41.781281 23387 raft_consensus.cc:2267] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:41.781539 23388 ts_tablet_manager.cc:1905] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:00:41.783596 23387 ts_tablet_manager.cc:1905] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:00:41.795352 23388 ts_tablet_manager.cc:1918] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 21:00:41.795768 23388 log.cc:1198] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-2-root/wals/feec74a233bb410f893279c914f9128f
I20250109 21:00:41.796420 23388 ts_tablet_manager.cc:1939] T feec74a233bb410f893279c914f9128f P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting consensus metadata
I20250109 21:00:41.797300 23387 ts_tablet_manager.cc:1918] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 21:00:41.797717 23387 log.cc:1198] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-1-root/wals/feec74a233bb410f893279c914f9128f
I20250109 21:00:41.798344 23387 ts_tablet_manager.cc:1939] T feec74a233bb410f893279c914f9128f P b4a83a60513d47558a1b164c8631a921: Deleting consensus metadata
I20250109 21:00:41.799271 22997 catalog_manager.cc:4872] TS 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721): tablet feec74a233bb410f893279c914f9128f (table dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]) successfully deleted
I20250109 21:00:41.800452 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:41.800448 23388 tablet_replica.cc:331] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939: stopping tablet replica
I20250109 21:00:41.801005 23387 tablet_replica.cc:331] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921: stopping tablet replica
I20250109 21:00:41.801095 22994 catalog_manager.cc:4872] TS b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647): tablet feec74a233bb410f893279c914f9128f (table dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]) successfully deleted
I20250109 21:00:41.801416 23388 raft_consensus.cc:2238] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:41.801908 23387 raft_consensus.cc:2238] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:41.801959 23388 raft_consensus.cc:2267] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:41.802467 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:41.802511 23387 raft_consensus.cc:2267] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:41.804421 23388 ts_tablet_manager.cc:1905] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:00:41.804711 23387 ts_tablet_manager.cc:1905] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:00:41.805364 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:41.807781 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:00:41.807971 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:41.808209 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:00:41.808359 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:41.818748 23388 ts_tablet_manager.cc:1918] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 21:00:41.818748 23387 ts_tablet_manager.cc:1918] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 21:00:41.819343 23388 log.cc:1198] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-2-root/wals/9ca95e9dff61489f98261bfd76c4cd3a
I20250109 21:00:41.820048 23388 ts_tablet_manager.cc:1939] T 9ca95e9dff61489f98261bfd76c4cd3a P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting consensus metadata
I20250109 21:00:41.819464 23387 log.cc:1198] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-1-root/wals/9ca95e9dff61489f98261bfd76c4cd3a
I20250109 21:00:41.821535 23387 ts_tablet_manager.cc:1939] T 9ca95e9dff61489f98261bfd76c4cd3a P b4a83a60513d47558a1b164c8631a921: Deleting consensus metadata
I20250109 21:00:41.822817 22997 catalog_manager.cc:4872] TS 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721): tablet 9ca95e9dff61489f98261bfd76c4cd3a (table dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]) successfully deleted
I20250109 21:00:41.824028 22994 catalog_manager.cc:4872] TS b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647): tablet 9ca95e9dff61489f98261bfd76c4cd3a (table dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]) successfully deleted
I20250109 21:00:41.857765 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.132:44459
I20250109 21:00:41.857852 23460 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.132:44459 every 8 connection(s)
I20250109 21:00:41.861972 17466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:00:41.868093 23464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:00:41.870369 23465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:41.871798 17466 server_base.cc:1034] running on GCE node
W20250109 21:00:41.872747 23467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:00:41.873844 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250109 21:00:41.874079 17466 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250109 21:00:41.874326 17466 hybrid_clock.cc:648] HybridClock initialized: now 1736456441874306 us; error 0 us; skew 500 ppm
I20250109 21:00:41.874873 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 21:00:41.874902 23461 heartbeater.cc:346] Connected to a master server at 127.17.14.190:38833
I20250109 21:00:41.875377 23461 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:41.876133 23461 heartbeater.cc:510] Master 127.17.14.190:38833 requested a full tablet report, sending...
I20250109 21:00:41.877854 17466 webserver.cc:458] Webserver started at http://127.17.14.129:42859/ using document root <none> and password file <none>
I20250109 21:00:41.877910 23010 ts_manager.cc:194] Registered new tserver with Master: 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132:44459)
I20250109 21:00:41.878494 17466 fs_manager.cc:362] Metadata directory not provided
I20250109 21:00:41.878746 17466 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250109 21:00:41.879305 23010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39230
I20250109 21:00:41.882333 17466 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20250109 21:00:41.885672 23472 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:00:41.886329 17466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:00:41.886579 17466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root
uuid: "06707233fdb442b79ca67eceabe36ebf"
format_stamp: "Formatted at 2025-01-09 21:00:38 on dist-test-slave-j54z"
I20250109 21:00:41.886823 17466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:00:41.900640 17466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:00:41.901754 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:00:41.903090 17466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:00:41.906296 23479 ts_tablet_manager.cc:542] Loading tablet metadata (0/8 complete)
I20250109 21:00:41.933990 23261 consensus_queue.cc:237] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
I20250109 21:00:41.946028 23185 raft_consensus.cc:1270] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:00:41.948141 23330 consensus_queue.cc:1035] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 21:00:41.952234 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20250109 21:00:41.952598 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.047s user 0.002s sys 0.000s
I20250109 21:00:41.952900 17466 ts_tablet_manager.cc:594] Registering tablets (0/8 complete)
I20250109 21:00:41.960258 23479 tablet_bootstrap.cc:492] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
W20250109 21:00:41.960922 23225 consensus_peers.cc:487] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 -> Peer 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132:44459): Couldn't send request to peer 19622fe17ad2450fb04e59fff92d9f5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d7958099f434d9baa13af9f2a891c4c. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:41.963508 23330 raft_consensus.cc:2949] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:41.966956 23185 raft_consensus.cc:2949] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:41.975498 22997 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1d7958099f434d9baa13af9f2a891c4c with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20250109 21:00:41.979975 23010 catalog_manager.cc:5526] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 reported cstate change: config changed from index -1 to 2, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New cstate: current_term: 1 leader_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:41.986274 23479 tablet_bootstrap.cc:492] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:41.987282 23479 tablet_bootstrap.cc:492] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf: Bootstrap complete.
I20250109 21:00:41.987895 23479 ts_tablet_manager.cc:1397] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.011s
I20250109 21:00:41.990227 23479 raft_consensus.cc:357] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:41.990928 23479 raft_consensus.cc:738] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:41.991554 23479 consensus_queue.cc:260] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:41.995553 23479 ts_tablet_manager.cc:1428] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250109 21:00:41.996356 23479 tablet_bootstrap.cc:492] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:42.000487 17466 ts_tablet_manager.cc:610] Registered 8 tablets
I20250109 21:00:42.000854 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.048s user 0.043s sys 0.002s
I20250109 21:00:42.009141 23479 tablet_bootstrap.cc:492] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:42.010200 23479 tablet_bootstrap.cc:492] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: Bootstrap complete.
I20250109 21:00:42.011018 23479 ts_tablet_manager.cc:1397] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20250109 21:00:42.013980 23479 raft_consensus.cc:357] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.014917 23479 raft_consensus.cc:738] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:42.015754 23479 consensus_queue.cc:260] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.018443 23479 ts_tablet_manager.cc:1428] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250109 21:00:42.019410 23479 tablet_bootstrap.cc:492] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:42.036370 23479 tablet_bootstrap.cc:492] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:42.037079 23479 tablet_bootstrap.cc:492] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf: Bootstrap complete.
I20250109 21:00:42.037698 23479 ts_tablet_manager.cc:1397] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.008s
I20250109 21:00:42.039783 23479 raft_consensus.cc:357] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.040304 23479 raft_consensus.cc:738] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:42.040920 23479 consensus_queue.cc:260] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.042624 23479 ts_tablet_manager.cc:1428] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:00:42.043219 23479 tablet_bootstrap.cc:492] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:42.052801 17466 rpc_server.cc:307] RPC server started. Bound to: 127.17.14.129:46465
I20250109 21:00:42.052930 23544 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.14.129:46465 every 8 connection(s)
I20250109 21:00:42.055083 23479 tablet_bootstrap.cc:492] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:42.055671 23479 tablet_bootstrap.cc:492] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: Bootstrap complete.
I20250109 21:00:42.056169 23479 ts_tablet_manager.cc:1397] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20250109 21:00:42.058207 23479 raft_consensus.cc:357] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.058946 23479 raft_consensus.cc:738] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:42.059768 23479 consensus_queue.cc:260] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.061177 23479 ts_tablet_manager.cc:1428] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:00:42.061829 23479 tablet_bootstrap.cc:492] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:42.068331 23545 heartbeater.cc:346] Connected to a master server at 127.17.14.190:38833
I20250109 21:00:42.068682 23545 heartbeater.cc:463] Registering TS with master...
I20250109 21:00:42.068570 23261 consensus_queue.cc:237] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
I20250109 21:00:42.069392 23545 heartbeater.cc:510] Master 127.17.14.190:38833 requested a full tablet report, sending...
I20250109 21:00:42.074450 23010 ts_manager.cc:194] Re-registered known tserver with Master: 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465)
I20250109 21:00:42.075506 23185 raft_consensus.cc:1270] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250109 21:00:42.075915 23225 consensus_peers.cc:487] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 -> Peer 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132:44459): Couldn't send request to peer 19622fe17ad2450fb04e59fff92d9f5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 883b7e05a84b48db811dbb6e7727b6bb. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:42.076853 23330 consensus_queue.cc:1035] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 21:00:42.078449 23479 tablet_bootstrap.cc:492] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:42.079339 23479 tablet_bootstrap.cc:492] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf: Bootstrap complete.
I20250109 21:00:42.079823 23479 ts_tablet_manager.cc:1397] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.005s
I20250109 21:00:42.081422 23479 raft_consensus.cc:357] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } }
I20250109 21:00:42.082099 23479 raft_consensus.cc:738] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:42.082825 23479 consensus_queue.cc:260] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } }
I20250109 21:00:42.083616 23010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39238
I20250109 21:00:42.084767 23479 ts_tablet_manager.cc:1428] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250109 21:00:42.085413 23479 tablet_bootstrap.cc:492] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:42.084924 23313 raft_consensus.cc:2949] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:42.086503 23545 heartbeater.cc:502] Master 127.17.14.190:38833 was elected leader, sending a full tablet report...
I20250109 21:00:42.089425 23185 raft_consensus.cc:2949] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:42.094774 22997 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 883b7e05a84b48db811dbb6e7727b6bb with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250109 21:00:42.101882 23479 tablet_bootstrap.cc:492] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:42.102684 23479 tablet_bootstrap.cc:492] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: Bootstrap complete.
I20250109 21:00:42.103324 23479 ts_tablet_manager.cc:1397] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20250109 21:00:42.103202 23009 catalog_manager.cc:5526] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 reported cstate change: config changed from index -1 to 2, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New cstate: current_term: 1 leader_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:42.105679 23479 raft_consensus.cc:357] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.106330 23479 raft_consensus.cc:738] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:42.106844 23479 consensus_queue.cc:260] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.108243 23479 ts_tablet_manager.cc:1428] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:00:42.108953 23479 tablet_bootstrap.cc:492] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:42.120471 23479 tablet_bootstrap.cc:492] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250109 21:00:42.121173 23479 tablet_bootstrap.cc:492] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Bootstrap complete.
I20250109 21:00:42.121722 23479 ts_tablet_manager.cc:1397] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20250109 21:00:42.122378 23379 raft_consensus.cc:491] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06707233fdb442b79ca67eceabe36ebf)
I20250109 21:00:42.122861 23379 raft_consensus.cc:513] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.123716 23479 raft_consensus.cc:357] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.124589 23479 raft_consensus.cc:738] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:42.124819 23379 leader_election.cc:290] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:42.125245 23479 consensus_queue.cc:260] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.125792 23313 raft_consensus.cc:491] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06707233fdb442b79ca67eceabe36ebf)
I20250109 21:00:42.126243 23313 raft_consensus.cc:513] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.127183 23479 ts_tablet_manager.cc:1428] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:00:42.127187 23261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8b8c42dee49a49b83fdcf1cd89a9a" candidate_uuid: "b4a83a60513d47558a1b164c8631a921" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" is_pre_election: true
I20250109 21:00:42.127869 23261 raft_consensus.cc:2463] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a83a60513d47558a1b164c8631a921 in term 1.
I20250109 21:00:42.128124 23479 tablet_bootstrap.cc:492] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Bootstrap starting.
I20250109 21:00:42.128086 23313 leader_election.cc:290] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647)
I20250109 21:00:42.129063 23153 leader_election.cc:304] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b2f500b0ae9460d8ea8defcb24c4939, b4a83a60513d47558a1b164c8631a921; no voters:
I20250109 21:00:42.130077 23379 raft_consensus.cc:2798] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250109 21:00:42.130451 23379 raft_consensus.cc:491] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 06707233fdb442b79ca67eceabe36ebf)
I20250109 21:00:42.130781 23379 raft_consensus.cc:3054] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:42.131402 23185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8b8c42dee49a49b83fdcf1cd89a9a" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b4a83a60513d47558a1b164c8631a921" is_pre_election: true
I20250109 21:00:42.132840 23187 consensus_queue.cc:237] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
I20250109 21:00:42.139971 23479 tablet_bootstrap.cc:492] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250109 21:00:42.141113 23479 tablet_bootstrap.cc:492] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Bootstrap complete.
I20250109 21:00:42.141912 23479 ts_tablet_manager.cc:1397] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20250109 21:00:42.143069 23379 raft_consensus.cc:513] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.144670 23185 raft_consensus.cc:2388] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b2f500b0ae9460d8ea8defcb24c4939 in current term 2: Already voted for candidate b4a83a60513d47558a1b164c8631a921 in this term.
I20250109 21:00:42.144392 23479 raft_consensus.cc:357] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.145401 23479 raft_consensus.cc:738] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06707233fdb442b79ca67eceabe36ebf, State: Initialized, Role: FOLLOWER
I20250109 21:00:42.146118 23479 consensus_queue.cc:260] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.148423 23479 ts_tablet_manager.cc:1428] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:00:42.149060 23379 leader_election.cc:290] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [CANDIDATE]: Term 2 election: Requested vote from peers 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465), 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:42.150868 23261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8b8c42dee49a49b83fdcf1cd89a9a" candidate_uuid: "b4a83a60513d47558a1b164c8631a921" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b2f500b0ae9460d8ea8defcb24c4939"
I20250109 21:00:42.151674 23261 raft_consensus.cc:3054] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:42.157271 23260 raft_consensus.cc:1270] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:00:42.158960 23261 raft_consensus.cc:2463] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a83a60513d47558a1b164c8631a921 in term 2.
I20250109 21:00:42.158939 23556 consensus_queue.cc:1035] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250109 21:00:42.160151 23153 leader_election.cc:304] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b2f500b0ae9460d8ea8defcb24c4939, b4a83a60513d47558a1b164c8631a921; no voters:
I20250109 21:00:42.161689 23379 raft_consensus.cc:2798] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 21:00:42.164111 23379 raft_consensus.cc:695] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 2 LEADER]: Becoming Leader. State: Replica: b4a83a60513d47558a1b164c8631a921, State: Running, Role: LEADER
I20250109 21:00:42.164968 23379 consensus_queue.cc:237] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
I20250109 21:00:42.166157 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8b8c42dee49a49b83fdcf1cd89a9a" candidate_uuid: "b4a83a60513d47558a1b164c8631a921" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf"
I20250109 21:00:42.166536 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8b8c42dee49a49b83fdcf1cd89a9a" candidate_uuid: "b4a83a60513d47558a1b164c8631a921" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf" is_pre_election: true
I20250109 21:00:42.166805 23517 raft_consensus.cc:3054] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:00:42.167804 23516 raft_consensus.cc:1270] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250109 21:00:42.170410 23151 consensus_peers.cc:487] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 -> Peer 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132:44459): Couldn't send request to peer 19622fe17ad2450fb04e59fff92d9f5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65a5f4dded1343c08e21702f21074a86. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:42.170888 23556 consensus_queue.cc:1035] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 21:00:42.174263 23517 raft_consensus.cc:2463] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a83a60513d47558a1b164c8631a921 in term 2.
I20250109 21:00:42.184947 23558 raft_consensus.cc:2949] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:42.186682 23261 raft_consensus.cc:2949] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:42.197793 22994 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 65a5f4dded1343c08e21702f21074a86 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250109 21:00:42.200107 23010 catalog_manager.cc:5526] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 reported cstate change: config changed from index -1 to 2, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New cstate: current_term: 1 leader_uuid: "b4a83a60513d47558a1b164c8631a921" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:42.206099 23516 raft_consensus.cc:2949] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:42.207934 23008 catalog_manager.cc:5526] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 reported cstate change: term changed from 1 to 2, leader changed from 06707233fdb442b79ca67eceabe36ebf (127.17.14.129) to b4a83a60513d47558a1b164c8631a921 (127.17.14.130). New cstate: current_term: 2 leader_uuid: "b4a83a60513d47558a1b164c8631a921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: UNKNOWN } } }
W20250109 21:00:42.219095 23058 auto_rebalancer.cc:254] failed to send replica move request: Invalid argument: Server with UUID 19622fe17ad2450fb04e59fff92d9f5e is already a member of the config. RaftConfig: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
W20250109 21:00:42.222582 23058 auto_rebalancer.cc:663] Could not move replica: Illegal state: 1d7958099f434d9baa13af9f2a891c4c: source replica b4a83a60513d47558a1b164c8631a921 does not have REPLACE attribute set
W20250109 21:00:42.222913 23058 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: 1d7958099f434d9baa13af9f2a891c4c: source replica b4a83a60513d47558a1b164c8631a921 does not have REPLACE attribute set
W20250109 21:00:42.226647 23058 auto_rebalancer.cc:663] Could not move replica: Illegal state: 65a5f4dded1343c08e21702f21074a86: source replica 6b2f500b0ae9460d8ea8defcb24c4939 does not have REPLACE attribute set
W20250109 21:00:42.226963 23058 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: 65a5f4dded1343c08e21702f21074a86: source replica 6b2f500b0ae9460d8ea8defcb24c4939 does not have REPLACE attribute set
I20250109 21:00:42.228617 23187 consensus_queue.cc:237] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } }
W20250109 21:00:42.230779 23058 auto_rebalancer.cc:663] Could not move replica: Illegal state: 883b7e05a84b48db811dbb6e7727b6bb: source replica 06707233fdb442b79ca67eceabe36ebf does not have REPLACE attribute set
W20250109 21:00:42.231104 23058 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: 883b7e05a84b48db811dbb6e7727b6bb: source replica 06707233fdb442b79ca67eceabe36ebf does not have REPLACE attribute set
I20250109 21:00:42.233181 23261 raft_consensus.cc:1270] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 21:00:42.233453 23516 raft_consensus.cc:1270] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 21:00:42.234575 23558 consensus_queue.cc:1035] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:42.235208 23379 consensus_queue.cc:1035] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:42.242411 23379 raft_consensus.cc:2949] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } }
I20250109 21:00:42.249609 22994 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 65a5f4dded1343c08e21702f21074a86 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 21:00:42.250762 23516 raft_consensus.cc:2949] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } }
I20250109 21:00:42.251154 23261 raft_consensus.cc:2949] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } }
I20250109 21:00:42.253855 23010 catalog_manager.cc:5526] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 reported cstate change: config changed from index 2 to 3, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) evicted. New cstate: current_term: 1 leader_uuid: "b4a83a60513d47558a1b164c8631a921" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: HEALTHY } } }
I20250109 21:00:42.272075 23426 tablet_service.cc:1514] Processing DeleteTablet for tablet 65a5f4dded1343c08e21702f21074a86 with delete_type TABLET_DATA_TOMBSTONED (TS 19622fe17ad2450fb04e59fff92d9f5e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40844
W20250109 21:00:42.274562 22995 catalog_manager.cc:4839] TS 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132:44459): delete failed for tablet 65a5f4dded1343c08e21702f21074a86 because the tablet was not found. No further retry: Not found: Tablet not found: 65a5f4dded1343c08e21702f21074a86
I20250109 21:00:42.453447 23566 ts_tablet_manager.cc:927] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e: Initiating tablet copy from peer 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:42.455000 23566 tablet_copy_client.cc:323] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.131:44721
I20250109 21:00:42.463219 23271 tablet_copy_service.cc:140] P 6b2f500b0ae9460d8ea8defcb24c4939: Received BeginTabletCopySession request for tablet 1d7958099f434d9baa13af9f2a891c4c from peer 19622fe17ad2450fb04e59fff92d9f5e ({username='slave'} at 127.0.0.1:35884)
I20250109 21:00:42.463712 23271 tablet_copy_service.cc:161] P 6b2f500b0ae9460d8ea8defcb24c4939: Beginning new tablet copy session on tablet 1d7958099f434d9baa13af9f2a891c4c from peer 19622fe17ad2450fb04e59fff92d9f5e at {username='slave'} at 127.0.0.1:35884: session id = 19622fe17ad2450fb04e59fff92d9f5e-1d7958099f434d9baa13af9f2a891c4c
I20250109 21:00:42.469376 23271 tablet_copy_source_session.cc:215] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 21:00:42.471829 23566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7958099f434d9baa13af9f2a891c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:42.486662 23566 tablet_copy_client.cc:806] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e: tablet copy: Starting download of 0 data blocks...
I20250109 21:00:42.487133 23566 tablet_copy_client.cc:670] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e: tablet copy: Starting download of 1 WAL segments...
I20250109 21:00:42.490099 23566 tablet_copy_client.cc:538] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 21:00:42.496299 23566 tablet_bootstrap.cc:492] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e: Bootstrap starting.
I20250109 21:00:42.511734 23566 tablet_bootstrap.cc:492] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:42.512374 23566 tablet_bootstrap.cc:492] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e: Bootstrap complete.
I20250109 21:00:42.512871 23566 ts_tablet_manager.cc:1397] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20250109 21:00:42.514859 23566 raft_consensus.cc:357] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
I20250109 21:00:42.515451 23566 raft_consensus.cc:738] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19622fe17ad2450fb04e59fff92d9f5e, State: Initialized, Role: LEARNER
I20250109 21:00:42.515921 23566 consensus_queue.cc:260] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
I20250109 21:00:42.517652 23461 heartbeater.cc:502] Master 127.17.14.190:38833 was elected leader, sending a full tablet report...
I20250109 21:00:42.517966 23566 ts_tablet_manager.cc:1428] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e: Time spent starting tablet: real 0.005s user 0.006s sys 0.002s
I20250109 21:00:42.519622 23271 tablet_copy_service.cc:342] P 6b2f500b0ae9460d8ea8defcb24c4939: Request end of tablet copy session 19622fe17ad2450fb04e59fff92d9f5e-1d7958099f434d9baa13af9f2a891c4c received from {username='slave'} at 127.0.0.1:35884
I20250109 21:00:42.519985 23271 tablet_copy_service.cc:434] P 6b2f500b0ae9460d8ea8defcb24c4939: ending tablet copy session 19622fe17ad2450fb04e59fff92d9f5e-1d7958099f434d9baa13af9f2a891c4c on tablet 1d7958099f434d9baa13af9f2a891c4c with peer 19622fe17ad2450fb04e59fff92d9f5e
I20250109 21:00:42.589812 23261 raft_consensus.cc:1270] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 2 FOLLOWER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 21:00:42.590929 23379 consensus_queue.cc:1035] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 21:00:42.599318 23516 raft_consensus.cc:1270] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 2 FOLLOWER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 21:00:42.601727 23558 consensus_queue.cc:1035] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 21:00:42.680595 23566 ts_tablet_manager.cc:927] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: Initiating tablet copy from peer 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721)
I20250109 21:00:42.681887 23566 tablet_copy_client.cc:323] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.131:44721
I20250109 21:00:42.683136 23271 tablet_copy_service.cc:140] P 6b2f500b0ae9460d8ea8defcb24c4939: Received BeginTabletCopySession request for tablet 883b7e05a84b48db811dbb6e7727b6bb from peer 19622fe17ad2450fb04e59fff92d9f5e ({username='slave'} at 127.0.0.1:35884)
I20250109 21:00:42.683565 23271 tablet_copy_service.cc:161] P 6b2f500b0ae9460d8ea8defcb24c4939: Beginning new tablet copy session on tablet 883b7e05a84b48db811dbb6e7727b6bb from peer 19622fe17ad2450fb04e59fff92d9f5e at {username='slave'} at 127.0.0.1:35884: session id = 19622fe17ad2450fb04e59fff92d9f5e-883b7e05a84b48db811dbb6e7727b6bb
I20250109 21:00:42.688510 23271 tablet_copy_source_session.cc:215] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 21:00:42.690752 23566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883b7e05a84b48db811dbb6e7727b6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:42.700394 23566 tablet_copy_client.cc:806] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: tablet copy: Starting download of 0 data blocks...
I20250109 21:00:42.700840 23566 tablet_copy_client.cc:670] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: tablet copy: Starting download of 1 WAL segments...
I20250109 21:00:42.703620 23566 tablet_copy_client.cc:538] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 21:00:42.708838 23566 tablet_bootstrap.cc:492] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: Bootstrap starting.
I20250109 21:00:42.722306 23566 tablet_bootstrap.cc:492] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:42.722966 23566 tablet_bootstrap.cc:492] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: Bootstrap complete.
I20250109 21:00:42.723462 23566 ts_tablet_manager.cc:1397] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.001s
I20250109 21:00:42.725672 23566 raft_consensus.cc:357] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
I20250109 21:00:42.726251 23566 raft_consensus.cc:738] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19622fe17ad2450fb04e59fff92d9f5e, State: Initialized, Role: LEARNER
I20250109 21:00:42.726696 23566 consensus_queue.cc:260] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
I20250109 21:00:42.728346 23566 ts_tablet_manager.cc:1428] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:00:42.729707 23271 tablet_copy_service.cc:342] P 6b2f500b0ae9460d8ea8defcb24c4939: Request end of tablet copy session 19622fe17ad2450fb04e59fff92d9f5e-883b7e05a84b48db811dbb6e7727b6bb received from {username='slave'} at 127.0.0.1:35884
I20250109 21:00:42.730016 23271 tablet_copy_service.cc:434] P 6b2f500b0ae9460d8ea8defcb24c4939: ending tablet copy session 19622fe17ad2450fb04e59fff92d9f5e-883b7e05a84b48db811dbb6e7727b6bb on tablet 883b7e05a84b48db811dbb6e7727b6bb with peer 19622fe17ad2450fb04e59fff92d9f5e
I20250109 21:00:42.965298 23436 raft_consensus.cc:1212] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250109 21:00:42.976325 23516 raft_consensus.cc:1270] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:00:42.976454 23517 raft_consensus.cc:1270] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:00:42.976504 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8b8c42dee49a49b83fdcf1cd89a9a" candidate_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06707233fdb442b79ca67eceabe36ebf" is_pre_election: true
I20250109 21:00:42.977725 23313 consensus_queue.cc:1035] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:00:42.978096 23227 leader_election.cc:304] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b2f500b0ae9460d8ea8defcb24c4939; no voters: 06707233fdb442b79ca67eceabe36ebf, b4a83a60513d47558a1b164c8631a921
I20250109 21:00:42.977998 23578 consensus_queue.cc:1035] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250109 21:00:42.979068 23579 raft_consensus.cc:2743] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 21:00:42.982828 23517 raft_consensus.cc:2949] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:42.983578 23518 raft_consensus.cc:2949] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:43.004951 23508 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca95e9dff61489f98261bfd76c4cd3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:00:41 UTC) from {username='slave'} at 127.0.0.1:33132
I20250109 21:00:43.005292 23507 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c787e0ed42459da54d45a435f34bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:00:41 UTC) from {username='slave'} at 127.0.0.1:33132
I20250109 21:00:43.005705 23506 tablet_service.cc:1514] Processing DeleteTablet for tablet de00f78308924bc7aeec065d410f6c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:00:41 UTC) from {username='slave'} at 127.0.0.1:33132
I20250109 21:00:43.006137 23505 tablet_service.cc:1514] Processing DeleteTablet for tablet feec74a233bb410f893279c914f9128f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:00:41 UTC) from {username='slave'} at 127.0.0.1:33132
I20250109 21:00:43.007054 23583 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:43.007787 23583 raft_consensus.cc:2238] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:43.008179 23583 pending_rounds.cc:62] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Trying to abort 1 pending ops.
I20250109 21:00:43.008379 23583 pending_rounds.cc:69] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112525580510294016 op_type: NO_OP noop_request { }
I20250109 21:00:43.008675 23583 raft_consensus.cc:2883] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250109 21:00:43.008968 23583 raft_consensus.cc:2267] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:43.011349 23583 ts_tablet_manager.cc:1905] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:00:43.021812 23583 ts_tablet_manager.cc:1918] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 21:00:43.022162 23583 log.cc:1198] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root/wals/9ca95e9dff61489f98261bfd76c4cd3a
I20250109 21:00:43.022780 23583 ts_tablet_manager.cc:1939] T 9ca95e9dff61489f98261bfd76c4cd3a P 06707233fdb442b79ca67eceabe36ebf: Deleting consensus metadata
I20250109 21:00:43.025298 23583 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:43.025242 22997 catalog_manager.cc:4872] TS 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465): tablet 9ca95e9dff61489f98261bfd76c4cd3a (table dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]) successfully deleted
I20250109 21:00:43.026263 23583 raft_consensus.cc:2238] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:43.026832 23583 raft_consensus.cc:2267] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:43.028766 23583 ts_tablet_manager.cc:1905] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:00:43.039263 23583 ts_tablet_manager.cc:1918] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 21:00:43.039589 23583 log.cc:1198] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root/wals/47c787e0ed42459da54d45a435f34bd9
I20250109 21:00:43.040285 23583 ts_tablet_manager.cc:1939] T 47c787e0ed42459da54d45a435f34bd9 P 06707233fdb442b79ca67eceabe36ebf: Deleting consensus metadata
I20250109 21:00:43.042395 23583 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:43.042403 22997 catalog_manager.cc:4872] TS 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465): tablet 47c787e0ed42459da54d45a435f34bd9 (table dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]) successfully deleted
I20250109 21:00:43.043424 23583 raft_consensus.cc:2238] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:43.043866 23583 raft_consensus.cc:2267] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:43.045717 23583 ts_tablet_manager.cc:1905] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:00:43.056210 23583 ts_tablet_manager.cc:1918] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 21:00:43.056516 23583 log.cc:1198] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root/wals/de00f78308924bc7aeec065d410f6c4f
I20250109 21:00:43.057036 23583 ts_tablet_manager.cc:1939] T de00f78308924bc7aeec065d410f6c4f P 06707233fdb442b79ca67eceabe36ebf: Deleting consensus metadata
I20250109 21:00:43.059149 23583 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:43.059206 22997 catalog_manager.cc:4872] TS 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465): tablet de00f78308924bc7aeec065d410f6c4f (table dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]) successfully deleted
I20250109 21:00:43.059989 23583 raft_consensus.cc:2238] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:43.060325 23583 pending_rounds.cc:62] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Trying to abort 1 pending ops.
I20250109 21:00:43.060467 23583 pending_rounds.cc:69] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112525580166098944 op_type: NO_OP noop_request { }
I20250109 21:00:43.060711 23583 raft_consensus.cc:2883] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250109 21:00:43.060950 23583 raft_consensus.cc:2267] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:43.062819 23583 ts_tablet_manager.cc:1905] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:00:43.071528 23583 ts_tablet_manager.cc:1918] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 21:00:43.071801 23583 log.cc:1198] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root/wals/feec74a233bb410f893279c914f9128f
I20250109 21:00:43.072331 23583 ts_tablet_manager.cc:1939] T feec74a233bb410f893279c914f9128f P 06707233fdb442b79ca67eceabe36ebf: Deleting consensus metadata
I20250109 21:00:43.074402 22997 catalog_manager.cc:4872] TS 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465): tablet feec74a233bb410f893279c914f9128f (table dugtrio [id=88b0cccc86a04d20a17ed97080e4164b]) successfully deleted
I20250109 21:00:43.204234 23436 raft_consensus.cc:1212] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250109 21:00:43.255728 23187 consensus_queue.cc:237] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
I20250109 21:00:43.261173 23261 raft_consensus.cc:1270] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 2 FOLLOWER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250109 21:00:43.261270 23151 consensus_peers.cc:487] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 -> Peer 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132:44459): Couldn't send request to peer 19622fe17ad2450fb04e59fff92d9f5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37d8b8c42dee49a49b83fdcf1cd89a9a. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:43.261341 23517 raft_consensus.cc:1270] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 2 FOLLOWER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250109 21:00:43.262749 23584 consensus_queue.cc:1035] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:43.263447 23585 consensus_queue.cc:1035] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:43.269625 23558 raft_consensus.cc:2949] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:43.271030 23517 raft_consensus.cc:2949] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:43.271304 23261 raft_consensus.cc:2949] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:43.281587 23009 catalog_manager.cc:5526] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 reported cstate change: config changed from index -1 to 3, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New cstate: current_term: 2 leader_uuid: "b4a83a60513d47558a1b164c8631a921" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250109 21:00:43.283818 23058 auto_rebalancer.cc:254] failed to send replica move request: Invalid argument: Server with UUID 19622fe17ad2450fb04e59fff92d9f5e is already a member of the config. RaftConfig: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
W20250109 21:00:43.291741 23058 auto_rebalancer.cc:663] Could not move replica: Illegal state: 883b7e05a84b48db811dbb6e7727b6bb: source replica 06707233fdb442b79ca67eceabe36ebf does not have REPLACE attribute set
W20250109 21:00:43.292021 23058 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: 883b7e05a84b48db811dbb6e7727b6bb: source replica 06707233fdb442b79ca67eceabe36ebf does not have REPLACE attribute set
W20250109 21:00:43.300125 23058 auto_rebalancer.cc:663] Could not move replica: Illegal state: 1d7958099f434d9baa13af9f2a891c4c: source replica b4a83a60513d47558a1b164c8631a921 does not have REPLACE attribute set
W20250109 21:00:43.300390 23058 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: 1d7958099f434d9baa13af9f2a891c4c: source replica b4a83a60513d47558a1b164c8631a921 does not have REPLACE attribute set
I20250109 21:00:43.437542 23579 raft_consensus.cc:1059] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939: attempting to promote NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e to VOTER
I20250109 21:00:43.439303 23579 consensus_queue.cc:237] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } }
I20250109 21:00:43.445731 23185 raft_consensus.cc:1270] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 21:00:43.446141 23436 raft_consensus.cc:1270] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e [term 1 LEARNER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 21:00:43.446631 23517 raft_consensus.cc:1270] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 21:00:43.447412 23579 consensus_queue.cc:1035] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:43.448196 23313 consensus_queue.cc:1035] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250109 21:00:43.449368 23578 consensus_queue.cc:1035] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:43.462229 23579 raft_consensus.cc:2949] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:43.464150 23436 raft_consensus.cc:2949] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:43.474215 23185 raft_consensus.cc:2949] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:43.482268 23517 raft_consensus.cc:2949] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:43.487864 23008 catalog_manager.cc:5526] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e reported cstate change: config changed from index 2 to 3, 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:43.497033 23261 consensus_queue.cc:237] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } }
I20250109 21:00:43.505548 23187 raft_consensus.cc:1270] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 21:00:43.508028 23517 raft_consensus.cc:1270] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250109 21:00:43.509660 23313 consensus_queue.cc:1035] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:43.510284 23595 consensus_queue.cc:1035] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250109 21:00:43.516172 23261 consensus_queue.cc:237] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } }
I20250109 21:00:43.519026 23313 raft_consensus.cc:2949] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } }
I20250109 21:00:43.522305 23517 raft_consensus.cc:2949] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } }
I20250109 21:00:43.529178 22997 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 883b7e05a84b48db811dbb6e7727b6bb with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 21:00:43.530069 23436 raft_consensus.cc:1270] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e [term 1 FOLLOWER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250109 21:00:43.532552 23313 consensus_queue.cc:1035] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250109 21:00:43.534286 23186 raft_consensus.cc:1270] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250109 21:00:43.534662 23010 catalog_manager.cc:5526] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 reported cstate change: config changed from index 2 to 3, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) evicted. New cstate: current_term: 1 leader_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:43.536006 23595 consensus_queue.cc:1035] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:43.540925 23595 raft_consensus.cc:2949] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06707233fdb442b79ca67eceabe36ebf (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:43.548365 23436 raft_consensus.cc:2949] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06707233fdb442b79ca67eceabe36ebf (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:43.559782 23426 tablet_service.cc:1514] Processing DeleteTablet for tablet 883b7e05a84b48db811dbb6e7727b6bb with delete_type TABLET_DATA_TOMBSTONED (TS 19622fe17ad2450fb04e59fff92d9f5e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40844
I20250109 21:00:43.562183 23185 raft_consensus.cc:2949] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 06707233fdb442b79ca67eceabe36ebf (127.17.14.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:43.567857 23009 catalog_manager.cc:5526] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e reported cstate change: config changed from index 3 to 4, VOTER 06707233fdb442b79ca67eceabe36ebf (127.17.14.129) evicted. New cstate: current_term: 1 leader_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:43.570664 22997 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 1d7958099f434d9baa13af9f2a891c4c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 21:00:43.573784 23187 raft_consensus.cc:2949] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } }
I20250109 21:00:43.583071 23505 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d7958099f434d9baa13af9f2a891c4c with delete_type TABLET_DATA_TOMBSTONED (TS 06707233fdb442b79ca67eceabe36ebf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33132
I20250109 21:00:43.586653 23599 tablet_replica.cc:331] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: stopping tablet replica
I20250109 21:00:43.587797 23599 raft_consensus.cc:2238] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e [term 1 LEARNER]: Raft consensus shutting down.
I20250109 21:00:43.588727 23599 raft_consensus.cc:2267] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e [term 1 LEARNER]: Raft consensus is shut down!
I20250109 21:00:43.590147 23600 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:43.591110 23600 raft_consensus.cc:2238] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:43.591848 23600 raft_consensus.cc:2267] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:43.592280 23599 ts_tablet_manager.cc:1905] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250109 21:00:43.598376 23600 ts_tablet_manager.cc:1905] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250109 21:00:43.605163 23599 ts_tablet_manager.cc:1918] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20250109 21:00:43.605621 23599 log.cc:1198] T 883b7e05a84b48db811dbb6e7727b6bb P 19622fe17ad2450fb04e59fff92d9f5e: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-3-root/wals/883b7e05a84b48db811dbb6e7727b6bb
I20250109 21:00:43.606943 22995 catalog_manager.cc:4872] TS 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132:44459): tablet 883b7e05a84b48db811dbb6e7727b6bb (table test-workload [id=15e8613ceb444b6893261fdb1701be76]) successfully deleted
I20250109 21:00:43.611083 23600 ts_tablet_manager.cc:1918] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250109 21:00:43.611438 23600 log.cc:1198] T 1d7958099f434d9baa13af9f2a891c4c P 06707233fdb442b79ca67eceabe36ebf: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-0-root/wals/1d7958099f434d9baa13af9f2a891c4c
I20250109 21:00:43.612715 22997 catalog_manager.cc:4872] TS 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465): tablet 1d7958099f434d9baa13af9f2a891c4c (table test-workload [id=15e8613ceb444b6893261fdb1701be76]) successfully deleted
I20250109 21:00:43.656219 23601 ts_tablet_manager.cc:927] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e: Initiating tablet copy from peer b4a83a60513d47558a1b164c8631a921 (127.17.14.130:39647)
I20250109 21:00:43.657691 23601 tablet_copy_client.cc:323] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e: tablet copy: Beginning tablet copy session from remote peer at address 127.17.14.130:39647
I20250109 21:00:43.668088 23197 tablet_copy_service.cc:140] P b4a83a60513d47558a1b164c8631a921: Received BeginTabletCopySession request for tablet 37d8b8c42dee49a49b83fdcf1cd89a9a from peer 19622fe17ad2450fb04e59fff92d9f5e ({username='slave'} at 127.0.0.1:42824)
I20250109 21:00:43.668540 23197 tablet_copy_service.cc:161] P b4a83a60513d47558a1b164c8631a921: Beginning new tablet copy session on tablet 37d8b8c42dee49a49b83fdcf1cd89a9a from peer 19622fe17ad2450fb04e59fff92d9f5e at {username='slave'} at 127.0.0.1:42824: session id = 19622fe17ad2450fb04e59fff92d9f5e-37d8b8c42dee49a49b83fdcf1cd89a9a
I20250109 21:00:43.675521 23197 tablet_copy_source_session.cc:215] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 21:00:43.678445 23601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d8b8c42dee49a49b83fdcf1cd89a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:00:43.692243 23601 tablet_copy_client.cc:806] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e: tablet copy: Starting download of 0 data blocks...
I20250109 21:00:43.692809 23601 tablet_copy_client.cc:670] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e: tablet copy: Starting download of 1 WAL segments...
I20250109 21:00:43.695956 23601 tablet_copy_client.cc:538] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 21:00:43.701864 23601 tablet_bootstrap.cc:492] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e: Bootstrap starting.
I20250109 21:00:43.720481 23601 tablet_bootstrap.cc:492] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:00:43.721251 23601 tablet_bootstrap.cc:492] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e: Bootstrap complete.
I20250109 21:00:43.721923 23601 ts_tablet_manager.cc:1397] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.004s
I20250109 21:00:43.724306 23601 raft_consensus.cc:357] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
I20250109 21:00:43.725176 23601 raft_consensus.cc:738] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 19622fe17ad2450fb04e59fff92d9f5e, State: Initialized, Role: LEARNER
I20250109 21:00:43.725792 23601 consensus_queue.cc:260] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
I20250109 21:00:43.728053 23601 ts_tablet_manager.cc:1428] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:00:43.729609 23197 tablet_copy_service.cc:342] P b4a83a60513d47558a1b164c8631a921: Request end of tablet copy session 19622fe17ad2450fb04e59fff92d9f5e-37d8b8c42dee49a49b83fdcf1cd89a9a received from {username='slave'} at 127.0.0.1:42824
I20250109 21:00:43.729943 23197 tablet_copy_service.cc:434] P b4a83a60513d47558a1b164c8631a921: ending tablet copy session 19622fe17ad2450fb04e59fff92d9f5e-37d8b8c42dee49a49b83fdcf1cd89a9a on tablet 37d8b8c42dee49a49b83fdcf1cd89a9a with peer 19622fe17ad2450fb04e59fff92d9f5e
I20250109 21:00:44.192173 23436 raft_consensus.cc:1212] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250109 21:00:44.576781 23606 raft_consensus.cc:1059] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921: attempting to promote NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e to VOTER
I20250109 21:00:44.580528 23606 consensus_queue.cc:237] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } }
I20250109 21:00:44.587307 23261 raft_consensus.cc:1270] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 2 FOLLOWER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:44.588135 23517 raft_consensus.cc:1270] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 2 FOLLOWER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:44.588872 23436 raft_consensus.cc:1270] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e [term 2 LEARNER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250109 21:00:44.590102 23585 consensus_queue.cc:1035] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250109 21:00:44.591290 23605 consensus_queue.cc:1035] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250109 21:00:44.592860 23585 consensus_queue.cc:1035] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:44.610548 23606 raft_consensus.cc:2949] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:44.612583 23261 raft_consensus.cc:2949] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:44.614356 23517 raft_consensus.cc:2949] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:44.625977 23010 catalog_manager.cc:5526] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 reported cstate change: config changed from index 3 to 4, 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b4a83a60513d47558a1b164c8631a921" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:44.625759 23436 raft_consensus.cc:2949] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:44.655869 23184 consensus_queue.cc:237] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } }
I20250109 21:00:44.660269 23436 raft_consensus.cc:1270] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e [term 2 FOLLOWER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 21:00:44.660825 23517 raft_consensus.cc:1270] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 2 FOLLOWER]: Refusing update from remote peer b4a83a60513d47558a1b164c8631a921: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 21:00:44.662144 23605 consensus_queue.cc:1035] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 21:00:44.662937 23606 consensus_queue.cc:1035] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 21:00:44.668593 23585 raft_consensus.cc:2949] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:44.670143 23517 raft_consensus.cc:2949] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:44.674597 23436 raft_consensus.cc:2949] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } } }
I20250109 21:00:44.676367 22994 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 37d8b8c42dee49a49b83fdcf1cd89a9a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250109 21:00:44.680840 23007 catalog_manager.cc:5526] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 reported cstate change: config changed from index 4 to 5, VOTER 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131) evicted. New cstate: current_term: 2 leader_uuid: "b4a83a60513d47558a1b164c8631a921" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250109 21:00:44.690634 23251 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d8b8c42dee49a49b83fdcf1cd89a9a with delete_type TABLET_DATA_TOMBSTONED (TS 6b2f500b0ae9460d8ea8defcb24c4939 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47434
I20250109 21:00:44.692930 23617 tablet_replica.cc:331] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939: stopping tablet replica
I20250109 21:00:44.693840 23617 raft_consensus.cc:2238] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:44.694626 23617 raft_consensus.cc:2267] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:44.697647 23617 ts_tablet_manager.cc:1905] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250109 21:00:44.708341 23617 ts_tablet_manager.cc:1918] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250109 21:00:44.708683 23617 log.cc:1198] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 6b2f500b0ae9460d8ea8defcb24c4939: Deleting WAL directory at /tmp/dist-test-tasknYvvDV/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736456270445890-17466-0/minicluster-data/ts-2-root/wals/37d8b8c42dee49a49b83fdcf1cd89a9a
I20250109 21:00:44.710291 22997 catalog_manager.cc:4872] TS 6b2f500b0ae9460d8ea8defcb24c4939 (127.17.14.131:44721): tablet 37d8b8c42dee49a49b83fdcf1cd89a9a (table test-workload [id=15e8613ceb444b6893261fdb1701be76]) successfully deleted
I20250109 21:00:45.688609 23261 consensus_queue.cc:237] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } }
W20250109 21:00:45.695966 23225 consensus_peers.cc:487] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 -> Peer 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132:44459): Couldn't send request to peer 19622fe17ad2450fb04e59fff92d9f5e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:45.697798 23187 raft_consensus.cc:1270] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250109 21:00:45.698316 23517 raft_consensus.cc:1270] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Refusing update from remote peer 6b2f500b0ae9460d8ea8defcb24c4939: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250109 21:00:45.699206 23619 consensus_queue.cc:1035] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:45.700162 23313 consensus_queue.cc:1035] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250109 21:00:45.707417 23313 raft_consensus.cc:2949] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:45.708591 23517 raft_consensus.cc:2949] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:45.709020 23187 raft_consensus.cc:2949] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } } }
I20250109 21:00:45.710628 17466 tablet_server.cc:178] TabletServer@127.17.14.129:46465 shutting down...
I20250109 21:00:45.726972 23009 catalog_manager.cc:5526] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 reported cstate change: config changed from index 3 to 4, NON_VOTER 19622fe17ad2450fb04e59fff92d9f5e (127.17.14.132) added. New cstate: current_term: 1 leader_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b2f500b0ae9460d8ea8defcb24c4939" member_type: VOTER last_known_addr { host: "127.17.14.131" port: 44721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06707233fdb442b79ca67eceabe36ebf" member_type: VOTER last_known_addr { host: "127.17.14.129" port: 46465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4a83a60513d47558a1b164c8631a921" member_type: VOTER last_known_addr { host: "127.17.14.130" port: 39647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19622fe17ad2450fb04e59fff92d9f5e" member_type: NON_VOTER last_known_addr { host: "127.17.14.132" port: 44459 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250109 21:00:45.769013 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:45.770145 17466 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:45.770854 17466 raft_consensus.cc:2238] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:45.771548 17466 raft_consensus.cc:2267] T 65a5f4dded1343c08e21702f21074a86 P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:45.774044 17466 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:45.774580 17466 raft_consensus.cc:2238] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:45.775139 17466 raft_consensus.cc:2267] T 883b7e05a84b48db811dbb6e7727b6bb P 06707233fdb442b79ca67eceabe36ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:45.777932 17466 tablet_replica.cc:331] stopping tablet replica
I20250109 21:00:45.778430 17466 raft_consensus.cc:2238] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:45.779132 17466 raft_consensus.cc:2267] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 06707233fdb442b79ca67eceabe36ebf [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:45.793969 17466 tablet_server.cc:195] TabletServer@127.17.14.129:46465 shutdown complete.
W20250109 21:00:45.801282 23153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.14.129:46465: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20250109 21:00:45.805195 23153 consensus_peers.cc:487] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 -> Peer 06707233fdb442b79ca67eceabe36ebf (127.17.14.129:46465): Couldn't send request to peer 06707233fdb442b79ca67eceabe36ebf. Status: Network error: Client connection negotiation failed: client connection to 127.17.14.129:46465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:00:45.823271 17466 tablet_server.cc:178] TabletServer@127.17.14.130:0 shutting down...
I20250109 21:00:45.848814 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:45.850961 17466 tablet_replica.cc:331] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921: stopping tablet replica
I20250109 21:00:45.851770 17466 raft_consensus.cc:2238] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 2 LEADER]: Raft consensus shutting down.
I20250109 21:00:45.852615 17466 raft_consensus.cc:2267] T 37d8b8c42dee49a49b83fdcf1cd89a9a P b4a83a60513d47558a1b164c8631a921 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:45.855216 17466 tablet_replica.cc:331] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921: stopping tablet replica
I20250109 21:00:45.855688 17466 raft_consensus.cc:2238] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:45.856582 17466 raft_consensus.cc:2267] T 65a5f4dded1343c08e21702f21074a86 P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:45.858938 17466 tablet_replica.cc:331] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921: stopping tablet replica
I20250109 21:00:45.859550 17466 raft_consensus.cc:2238] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:45.860307 17466 raft_consensus.cc:2267] T 883b7e05a84b48db811dbb6e7727b6bb P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:45.862764 17466 tablet_replica.cc:331] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921: stopping tablet replica
I20250109 21:00:45.863197 17466 raft_consensus.cc:2238] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:45.863830 17466 raft_consensus.cc:2267] T 1d7958099f434d9baa13af9f2a891c4c P b4a83a60513d47558a1b164c8631a921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:45.890570 17466 tablet_server.cc:195] TabletServer@127.17.14.130:0 shutdown complete.
I20250109 21:00:45.916409 17466 tablet_server.cc:178] TabletServer@127.17.14.131:0 shutting down...
W20250109 21:00:45.922940 23058 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250109 21:00:45.923224 23058 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250109 21:00:45.938989 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:45.939675 17466 tablet_replica.cc:331] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939: stopping tablet replica
I20250109 21:00:45.940155 17466 raft_consensus.cc:2238] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:45.940644 17466 raft_consensus.cc:2267] T 65a5f4dded1343c08e21702f21074a86 P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:45.942488 17466 tablet_replica.cc:331] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939: stopping tablet replica
I20250109 21:00:45.942888 17466 raft_consensus.cc:2238] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:45.943547 17466 raft_consensus.cc:2267] T 1d7958099f434d9baa13af9f2a891c4c P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:45.945256 17466 tablet_replica.cc:331] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939: stopping tablet replica
I20250109 21:00:45.945802 17466 raft_consensus.cc:2238] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:45.946586 17466 raft_consensus.cc:2267] T 883b7e05a84b48db811dbb6e7727b6bb P 6b2f500b0ae9460d8ea8defcb24c4939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:45.959759 17466 tablet_server.cc:195] TabletServer@127.17.14.131:0 shutdown complete.
I20250109 21:00:45.978863 17466 tablet_server.cc:178] TabletServer@127.17.14.132:0 shutting down...
I20250109 21:00:45.997822 17466 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250109 21:00:45.998466 17466 tablet_replica.cc:331] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e: stopping tablet replica
I20250109 21:00:45.998950 17466 raft_consensus.cc:2238] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:45.999450 17466 raft_consensus.cc:2267] T 37d8b8c42dee49a49b83fdcf1cd89a9a P 19622fe17ad2450fb04e59fff92d9f5e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:46.001253 17466 tablet_replica.cc:331] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e: stopping tablet replica
I20250109 21:00:46.001760 17466 raft_consensus.cc:2238] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:00:46.002233 17466 raft_consensus.cc:2267] T 1d7958099f434d9baa13af9f2a891c4c P 19622fe17ad2450fb04e59fff92d9f5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:46.022361 17466 tablet_server.cc:195] TabletServer@127.17.14.132:0 shutdown complete.
I20250109 21:00:46.033972 17466 master.cc:537] Master@127.17.14.190:38833 shutting down...
I20250109 21:00:46.051045 17466 raft_consensus.cc:2238] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:00:46.051512 17466 raft_consensus.cc:2267] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:00:46.051787 17466 tablet_replica.cc:331] T 00000000000000000000000000000000 P bd6dfd4c014744338f9aa894a45e0eed: stopping tablet replica
I20250109 21:00:46.070066 17466 master.cc:559] Master@127.17.14.190:38833 shutdown complete.
[ OK ] AutoRebalancerTest.TestDeletedTables (8086 ms)
[----------] 14 tests from AutoRebalancerTest (175503 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (175503 ms total)
[ PASSED ] 12 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] AutoRebalancerTest.TestMaxMovesPerServer
[ FAILED ] 1 test, listed below:
[ FAILED ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
1 FAILED TEST
I20250109 21:00:46.112072 17466 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:594: suppressed but not reported on 7 messages since previous log ~4 seconds ago
I20250109 21:00:46.112329 17466 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:542: suppressed but not reported on 7 messages since previous log ~4 seconds ago
I20250109 21:00:46.112504 17466 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:4615: suppressed but not reported on 14 messages since previous log ~4 seconds ago
I20250109 21:00:46.112681 17466 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:5204: suppressed but not reported on 67 messages since previous log ~40 seconds ago
I20250109 21:00:46.113070 17466 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2367: suppressed but not reported on 51 messages since previous log ~20 seconds ago
I20250109 21:00:46.113243 17466 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 1 messages since previous log ~0 seconds ago