[==========] Running 22 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 22 tests from RemoteKsckTest
[ RUN ] RemoteKsckTest.TestClusterOk
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250115 02:52:05.961237 2193 test_util.cc:274] Using random seed: -1018345061
I20250115 02:52:05.964908 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:45827,127.2.36.125:42201,127.2.36.124:45025
I20250115 02:52:05.966393 2193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 02:52:05.967429 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:05.983580 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:05.984376 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:05.985110 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:05.986034 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:07.086683 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:07.086972 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:07.087172 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909527087148 us; error 0 us; skew 500 ppm
I20250115 02:52:07.087996 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:07.097198 2193 webserver.cc:458] Webserver started at http://127.2.36.126:39041/ using document root <none> and password file <none>
I20250115 02:52:07.098042 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:07.098271 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:07.098675 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:07.104969 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "c977b7e3ed80457ab586c6445e72da06"
format_stamp: "Formatted at 2025-01-15 02:52:07 on dist-test-slave-1lv2"
I20250115 02:52:07.112497 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20250115 02:52:07.117520 2208 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:07.118599 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250115 02:52:07.118952 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "c977b7e3ed80457ab586c6445e72da06"
format_stamp: "Formatted at 2025-01-15 02:52:07 on dist-test-slave-1lv2"
I20250115 02:52:07.119292 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:07.189909 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:07.191458 2193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 02:52:07.191897 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:07.258291 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:45827
I20250115 02:52:07.258363 2259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:45827 every 8 connection(s)
I20250115 02:52:07.264004 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:07.264256 2260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 02:52:07.271155 2262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:07.273313 2263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:07.275349 2265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:07.276268 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:07.277086 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:07.277345 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:07.277549 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909527277530 us; error 0 us; skew 500 ppm
I20250115 02:52:07.278337 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:07.282331 2193 webserver.cc:458] Webserver started at http://127.2.36.125:39253/ using document root <none> and password file <none>
I20250115 02:52:07.283053 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:07.283303 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:07.283648 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:07.285076 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "cb1f33e2785e418b8324b1306d39a562"
format_stamp: "Formatted at 2025-01-15 02:52:07 on dist-test-slave-1lv2"
I20250115 02:52:07.291631 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250115 02:52:07.289572 2260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:07.298808 2270 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:07.299801 2193 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20250115 02:52:07.300164 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "cb1f33e2785e418b8324b1306d39a562"
format_stamp: "Formatted at 2025-01-15 02:52:07 on dist-test-slave-1lv2"
I20250115 02:52:07.300541 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:07.315021 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:07.316520 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:07.319751 2260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:07.322109 2210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.36.125:42201: connect: Connection refused (error 111)
W20250115 02:52:07.326007 2260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:42201: Network error: Client connection negotiation failed: client connection to 127.2.36.125:42201: connect: Connection refused (error 111)
I20250115 02:52:07.364887 2260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } attempt: 1
W20250115 02:52:07.369278 2260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:42201: Network error: Client connection negotiation failed: client connection to 127.2.36.125:42201: connect: Connection refused (error 111)
I20250115 02:52:07.380570 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:42201
I20250115 02:52:07.380723 2324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:42201 every 8 connection(s)
I20250115 02:52:07.385258 2325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:07.385928 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:07.391583 2325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:07.392820 2327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:07.393836 2328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:07.397243 2193 server_base.cc:1034] running on GCE node
W20250115 02:52:07.397720 2330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:07.398509 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:07.398706 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:07.398926 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909527398908 us; error 0 us; skew 500 ppm
I20250115 02:52:07.399474 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:07.402967 2193 webserver.cc:458] Webserver started at http://127.2.36.124:41137/ using document root <none> and password file <none>
I20250115 02:52:07.403468 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:07.403651 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:07.404207 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:07.405040 2325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:07.405504 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "7749ab69bfeb401b89687b764189ed9a"
format_stamp: "Formatted at 2025-01-15 02:52:07 on dist-test-slave-1lv2"
I20250115 02:52:07.411572 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250115 02:52:07.414775 2260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } attempt: 2
I20250115 02:52:07.416163 2337 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:07.416935 2325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:07.418370 2193 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250115 02:52:07.418723 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "7749ab69bfeb401b89687b764189ed9a"
format_stamp: "Formatted at 2025-01-15 02:52:07 on dist-test-slave-1lv2"
I20250115 02:52:07.419118 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250115 02:52:07.427188 2325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45025: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45025: connect: Connection refused (error 111)
I20250115 02:52:07.427381 2260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:07.431593 2260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45025: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45025: connect: Connection refused (error 111)
I20250115 02:52:07.448122 2325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } attempt: 1
W20250115 02:52:07.451829 2325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45025: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45025: connect: Connection refused (error 111)
I20250115 02:52:07.454629 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:07.455842 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:07.458001 2260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } attempt: 1
W20250115 02:52:07.461808 2260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45025: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45025: connect: Connection refused (error 111)
I20250115 02:52:07.500244 2325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } attempt: 2
W20250115 02:52:07.504283 2325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45025: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45025: connect: Connection refused (error 111)
I20250115 02:52:07.513207 2260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } attempt: 2
W20250115 02:52:07.518258 2260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45025: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45025: connect: Connection refused (error 111)
I20250115 02:52:07.519762 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:45025
I20250115 02:52:07.519866 2389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:45025 every 8 connection(s)
I20250115 02:52:07.523849 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:52:07.524070 2390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:07.529274 2390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:07.539652 2390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:07.547456 2390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:07.563745 2390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a: Bootstrap starting.
I20250115 02:52:07.569479 2390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:07.571341 2390 log.cc:826] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a: Log is configured to *not* fsync() on all Append() calls
I20250115 02:52:07.575721 2390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a: No bootstrap required, opened a new log
I20250115 02:52:07.594820 2325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } attempt: 3
I20250115 02:52:07.595736 2390 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } }
I20250115 02:52:07.596601 2390 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:07.596904 2390 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7749ab69bfeb401b89687b764189ed9a, State: Initialized, Role: FOLLOWER
I20250115 02:52:07.597891 2390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } }
I20250115 02:52:07.600559 2395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } } }
I20250115 02:52:07.601397 2395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:07.602825 2390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:07.611333 2325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562: Bootstrap starting.
I20250115 02:52:07.616837 2325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:07.620733 2260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } attempt: 3
I20250115 02:52:07.621397 2407 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250115 02:52:07.623270 2325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562: No bootstrap required, opened a new log
I20250115 02:52:07.625527 2325 raft_consensus.cc:357] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } }
I20250115 02:52:07.626256 2325 raft_consensus.cc:383] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:07.626546 2325 raft_consensus.cc:738] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb1f33e2785e418b8324b1306d39a562, State: Initialized, Role: FOLLOWER
I20250115 02:52:07.627230 2325 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } }
I20250115 02:52:07.630199 2408 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } } }
I20250115 02:52:07.631080 2408 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:07.631675 2325 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:07.634575 2260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06: Bootstrap starting.
I20250115 02:52:07.639176 2260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:07.644122 2260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06: No bootstrap required, opened a new log
I20250115 02:52:07.645995 2260 raft_consensus.cc:357] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } }
I20250115 02:52:07.646507 2260 raft_consensus.cc:383] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:07.646731 2260 raft_consensus.cc:738] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c977b7e3ed80457ab586c6445e72da06, State: Initialized, Role: FOLLOWER
I20250115 02:52:07.647245 2395 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:07.647588 2260 consensus_queue.cc:260] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } }
I20250115 02:52:07.647797 2395 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } }
I20250115 02:52:07.651930 2395 leader_election.cc:290] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c977b7e3ed80457ab586c6445e72da06 (127.2.36.126:45827), cb1f33e2785e418b8324b1306d39a562 (127.2.36.125:42201)
I20250115 02:52:07.653126 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7749ab69bfeb401b89687b764189ed9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb1f33e2785e418b8324b1306d39a562" is_pre_election: true
I20250115 02:52:07.653285 2419 sys_catalog.cc:455] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } } }
I20250115 02:52:07.654067 2419 sys_catalog.cc:458] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:07.654060 2300 raft_consensus.cc:2463] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7749ab69bfeb401b89687b764189ed9a in term 0.
I20250115 02:52:07.655207 2260 sys_catalog.cc:564] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:07.655591 2340 leader_election.cc:304] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7749ab69bfeb401b89687b764189ed9a, cb1f33e2785e418b8324b1306d39a562; no voters:
I20250115 02:52:07.656461 2395 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:07.656826 2395 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:07.656625 2235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7749ab69bfeb401b89687b764189ed9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c977b7e3ed80457ab586c6445e72da06" is_pre_election: true
I20250115 02:52:07.657146 2395 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:07.657367 2235 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7749ab69bfeb401b89687b764189ed9a in term 0.
I20250115 02:52:07.664206 2395 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } }
I20250115 02:52:07.666332 2395 leader_election.cc:290] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [CANDIDATE]: Term 1 election: Requested vote from peers c977b7e3ed80457ab586c6445e72da06 (127.2.36.126:45827), cb1f33e2785e418b8324b1306d39a562 (127.2.36.125:42201)
W20250115 02:52:07.667652 2421 catalog_manager.cc:1559] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:07.668026 2421 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:07.668630 2235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7749ab69bfeb401b89687b764189ed9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c977b7e3ed80457ab586c6445e72da06"
I20250115 02:52:07.669291 2235 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:07.668664 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7749ab69bfeb401b89687b764189ed9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb1f33e2785e418b8324b1306d39a562"
I20250115 02:52:07.670040 2300 raft_consensus.cc:3054] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:07.676380 2235 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7749ab69bfeb401b89687b764189ed9a in term 1.
I20250115 02:52:07.676369 2300 raft_consensus.cc:2463] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7749ab69bfeb401b89687b764189ed9a in term 1.
I20250115 02:52:07.677991 2340 leader_election.cc:304] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7749ab69bfeb401b89687b764189ed9a, c977b7e3ed80457ab586c6445e72da06; no voters:
I20250115 02:52:07.678851 2395 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:07.681260 2395 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 1 LEADER]: Becoming Leader. State: Replica: 7749ab69bfeb401b89687b764189ed9a, State: Running, Role: LEADER
I20250115 02:52:07.682384 2395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } }
I20250115 02:52:07.696218 2426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7749ab69bfeb401b89687b764189ed9a. Latest consensus state: current_term: 1 leader_uuid: "7749ab69bfeb401b89687b764189ed9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } } }
W20250115 02:52:07.697114 2214 tablet.cc:2367] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250115 02:52:07.697129 2426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:07.700348 2434 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:52:07.703599 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:52:07.703891 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:52:07.705746 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:07.707353 2436 catalog_manager.cc:1559] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:07.707664 2436 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:07.708243 2434 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250115 02:52:07.723435 2437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:07.726997 2438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:07.727881 2300 raft_consensus.cc:1270] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [term 1 FOLLOWER]: Refusing update from remote peer 7749ab69bfeb401b89687b764189ed9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:07.730047 2235 raft_consensus.cc:1270] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [term 1 FOLLOWER]: Refusing update from remote peer 7749ab69bfeb401b89687b764189ed9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:07.731931 2395 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:07.733598 2426 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 02:52:07.734930 2442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:07.740609 2408 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7749ab69bfeb401b89687b764189ed9a. Latest consensus state: current_term: 1 leader_uuid: "7749ab69bfeb401b89687b764189ed9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } } }
I20250115 02:52:07.741348 2408 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:07.742817 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:07.745183 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:07.745431 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:07.745630 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909527745611 us; error 0 us; skew 500 ppm
I20250115 02:52:07.746338 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:07.749876 2443 mvcc.cc:204] Tried to move back new op lower bound from 7114381425549582336 to 7114381425399787520. Current Snapshot: MvccSnapshot[applied={T|T < 7114381425549582336}]
I20250115 02:52:07.772114 2408 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7749ab69bfeb401b89687b764189ed9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } } }
I20250115 02:52:07.773003 2408 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:07.783923 2419 sys_catalog.cc:455] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7749ab69bfeb401b89687b764189ed9a. Latest consensus state: current_term: 1 leader_uuid: "7749ab69bfeb401b89687b764189ed9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } } }
I20250115 02:52:07.784760 2419 sys_catalog.cc:458] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:07.787065 2193 webserver.cc:458] Webserver started at http://127.2.36.65:42457/ using document root <none> and password file <none>
I20250115 02:52:07.787220 2426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7749ab69bfeb401b89687b764189ed9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } } }
I20250115 02:52:07.788316 2434 catalog_manager.cc:1348] Generated new cluster ID: 188667f56e094bbcba5725dee8e0370c
I20250115 02:52:07.788638 2434 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:52:07.788743 2426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:07.787776 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:07.789842 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:07.790246 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:07.791877 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "ee167c4ec6e1496389be5398e565c5a6"
format_stamp: "Formatted at 2025-01-15 02:52:07 on dist-test-slave-1lv2"
I20250115 02:52:07.799696 2451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7749ab69bfeb401b89687b764189ed9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } } }
I20250115 02:52:07.800580 2451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:07.805450 2419 sys_catalog.cc:455] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7749ab69bfeb401b89687b764189ed9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c977b7e3ed80457ab586c6445e72da06" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 45827 } } peers { permanent_uuid: "cb1f33e2785e418b8324b1306d39a562" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42201 } } peers { permanent_uuid: "7749ab69bfeb401b89687b764189ed9a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45025 } } }
I20250115 02:52:07.806196 2419 sys_catalog.cc:458] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:07.808054 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.007s sys 0.008s
I20250115 02:52:07.813542 2454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:07.814631 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250115 02:52:07.815106 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "ee167c4ec6e1496389be5398e565c5a6"
format_stamp: "Formatted at 2025-01-15 02:52:07 on dist-test-slave-1lv2"
I20250115 02:52:07.815446 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:07.825229 2434 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:52:07.827436 2434 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:52:07.853921 2434 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a: Generated new TSK 0
I20250115 02:52:07.855109 2434 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:52:07.865734 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:07.867515 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:07.870083 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:07.875123 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:07.875425 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:07.875722 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:07.875937 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:07.959849 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:44597
I20250115 02:52:07.959905 2516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:44597 every 8 connection(s)
I20250115 02:52:07.984165 2517 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45025
I20250115 02:52:07.984649 2517 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:07.984923 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:07.985930 2517 heartbeater.cc:509] Master 127.2.36.124:45025 requested a full tablet report, sending...
I20250115 02:52:07.990780 2355 ts_manager.cc:194] Registered new tserver with Master: ee167c4ec6e1496389be5398e565c5a6 (127.2.36.65:44597)
I20250115 02:52:07.994117 2518 heartbeater.cc:346] Connected to a master server at 127.2.36.125:42201
I20250115 02:52:07.994514 2518 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:07.995450 2518 heartbeater.cc:509] Master 127.2.36.125:42201 requested a full tablet report, sending...
I20250115 02:52:07.998452 2290 ts_manager.cc:194] Registered new tserver with Master: ee167c4ec6e1496389be5398e565c5a6 (127.2.36.65:44597)
I20250115 02:52:07.993849 2355 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50004
I20250115 02:52:08.005875 2520 heartbeater.cc:346] Connected to a master server at 127.2.36.126:45827
I20250115 02:52:08.006300 2520 heartbeater.cc:463] Registering TS with master...
W20250115 02:52:08.006260 2523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:08.007166 2520 heartbeater.cc:509] Master 127.2.36.126:45827 requested a full tablet report, sending...
W20250115 02:52:08.009011 2524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:08.010428 2225 ts_manager.cc:194] Registered new tserver with Master: ee167c4ec6e1496389be5398e565c5a6 (127.2.36.65:44597)
W20250115 02:52:08.014209 2527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:08.014360 2517 heartbeater.cc:501] Master 127.2.36.124:45025 was elected leader, sending a full tablet report...
I20250115 02:52:08.015623 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:08.016930 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:08.017212 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:08.017454 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909528017438 us; error 0 us; skew 500 ppm
I20250115 02:52:08.018218 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:08.021093 2193 webserver.cc:458] Webserver started at http://127.2.36.66:39509/ using document root <none> and password file <none>
I20250115 02:52:08.021775 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:08.022011 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:08.022344 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:08.023746 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "7dcb964e455f44da88719842b398c454"
format_stamp: "Formatted at 2025-01-15 02:52:08 on dist-test-slave-1lv2"
I20250115 02:52:08.030136 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250115 02:52:08.034119 2531 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:08.034945 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:52:08.035233 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "7dcb964e455f44da88719842b398c454"
format_stamp: "Formatted at 2025-01-15 02:52:08 on dist-test-slave-1lv2"
I20250115 02:52:08.035609 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:08.059747 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:08.060961 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:08.062279 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:08.065737 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:08.065997 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:08.066278 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:08.066479 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:08.168373 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:43203
I20250115 02:52:08.168447 2593 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:43203 every 8 connection(s)
I20250115 02:52:08.183940 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:08.201051 2600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:08.206032 2601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:08.207646 2595 heartbeater.cc:346] Connected to a master server at 127.2.36.125:42201
I20250115 02:52:08.208050 2595 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:08.208913 2595 heartbeater.cc:509] Master 127.2.36.125:42201 requested a full tablet report, sending...
I20250115 02:52:08.212074 2290 ts_manager.cc:194] Registered new tserver with Master: 7dcb964e455f44da88719842b398c454 (127.2.36.66:43203)
I20250115 02:52:08.213349 2594 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45025
I20250115 02:52:08.213712 2594 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:08.214639 2594 heartbeater.cc:509] Master 127.2.36.124:45025 requested a full tablet report, sending...
I20250115 02:52:08.222149 2355 ts_manager.cc:194] Registered new tserver with Master: 7dcb964e455f44da88719842b398c454 (127.2.36.66:43203)
I20250115 02:52:08.224018 2355 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50020
I20250115 02:52:08.232978 2596 heartbeater.cc:346] Connected to a master server at 127.2.36.126:45827
I20250115 02:52:08.233395 2596 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:08.234122 2596 heartbeater.cc:509] Master 127.2.36.126:45827 requested a full tablet report, sending...
W20250115 02:52:08.235186 2603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:08.236773 2225 ts_manager.cc:194] Registered new tserver with Master: 7dcb964e455f44da88719842b398c454 (127.2.36.66:43203)
I20250115 02:52:08.237903 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:08.238948 2594 heartbeater.cc:501] Master 127.2.36.124:45025 was elected leader, sending a full tablet report...
I20250115 02:52:08.239014 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:08.239423 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:08.239638 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909528239621 us; error 0 us; skew 500 ppm
I20250115 02:52:08.240417 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:08.243664 2193 webserver.cc:458] Webserver started at http://127.2.36.67:36747/ using document root <none> and password file <none>
I20250115 02:52:08.244227 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:08.244426 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:08.244696 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:08.246102 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "5daaa634d45349a9a9934cfa5fc76969"
format_stamp: "Formatted at 2025-01-15 02:52:08 on dist-test-slave-1lv2"
I20250115 02:52:08.252970 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20250115 02:52:08.256821 2608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:08.257627 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250115 02:52:08.257972 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "5daaa634d45349a9a9934cfa5fc76969"
format_stamp: "Formatted at 2025-01-15 02:52:08 on dist-test-slave-1lv2"
I20250115 02:52:08.258406 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterOk.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:08.275910 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:08.277424 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:08.279240 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:08.284940 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:08.285219 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:08.285516 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:08.285770 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250115 02:52:08.469930 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:44463
I20250115 02:52:08.470178 2670 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:44463 every 8 connection(s)
I20250115 02:52:08.507705 2671 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45025
I20250115 02:52:08.508113 2671 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:08.508972 2671 heartbeater.cc:509] Master 127.2.36.124:45025 requested a full tablet report, sending...
I20250115 02:52:08.518685 2355 ts_manager.cc:194] Registered new tserver with Master: 5daaa634d45349a9a9934cfa5fc76969 (127.2.36.67:44463)
I20250115 02:52:08.520536 2355 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50024
I20250115 02:52:08.533840 2672 heartbeater.cc:346] Connected to a master server at 127.2.36.125:42201
I20250115 02:52:08.535387 2671 heartbeater.cc:501] Master 127.2.36.124:45025 was elected leader, sending a full tablet report...
I20250115 02:52:08.535131 2672 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:08.538407 2672 heartbeater.cc:509] Master 127.2.36.125:42201 requested a full tablet report, sending...
I20250115 02:52:08.541471 2289 ts_manager.cc:194] Registered new tserver with Master: 5daaa634d45349a9a9934cfa5fc76969 (127.2.36.67:44463)
I20250115 02:52:08.557268 2674 heartbeater.cc:346] Connected to a master server at 127.2.36.126:45827
I20250115 02:52:08.557713 2674 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:08.558460 2674 heartbeater.cc:509] Master 127.2.36.126:45827 requested a full tablet report, sending...
I20250115 02:52:08.561240 2224 ts_manager.cc:194] Registered new tserver with Master: 5daaa634d45349a9a9934cfa5fc76969 (127.2.36.67:44463)
I20250115 02:52:08.561699 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.029027167s
I20250115 02:52:08.675194 2421 catalog_manager.cc:1260] Loaded cluster ID: 188667f56e094bbcba5725dee8e0370c
I20250115 02:52:08.675626 2421 catalog_manager.cc:1553] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562: loading cluster ID for follower catalog manager: success
I20250115 02:52:08.682487 2421 catalog_manager.cc:1575] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562: acquiring CA information for follower catalog manager: success
I20250115 02:52:08.688089 2421 catalog_manager.cc:1603] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:08.713850 2436 catalog_manager.cc:1260] Loaded cluster ID: 188667f56e094bbcba5725dee8e0370c
I20250115 02:52:08.714190 2436 catalog_manager.cc:1553] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06: loading cluster ID for follower catalog manager: success
I20250115 02:52:08.725728 2436 catalog_manager.cc:1575] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06: acquiring CA information for follower catalog manager: success
I20250115 02:52:08.649267 2355 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50028:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 02:52:08.749959 2436 catalog_manager.cc:1603] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250115 02:52:08.755892 2355 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:52:08.941567 2636 tablet_service.cc:1467] Processing CreateTablet for tablet 2260834807694f68b5b3f079e647e43b (DEFAULT_TABLE table=ksck-test-table [id=8c9e80327be34168a9ca0af3ed81e383]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:08.942427 2634 tablet_service.cc:1467] Processing CreateTablet for tablet a09692529c094da9bc48dab627698e3b (DEFAULT_TABLE table=ksck-test-table [id=8c9e80327be34168a9ca0af3ed81e383]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:08.943670 2636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2260834807694f68b5b3f079e647e43b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:08.943410 2635 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6cf7bdd47644f29fe15f5bddd380a7 (DEFAULT_TABLE table=ksck-test-table [id=8c9e80327be34168a9ca0af3ed81e383]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:08.948876 2634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a09692529c094da9bc48dab627698e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:08.955883 2635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6cf7bdd47644f29fe15f5bddd380a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:08.978272 2480 tablet_service.cc:1467] Processing CreateTablet for tablet a09692529c094da9bc48dab627698e3b (DEFAULT_TABLE table=ksck-test-table [id=8c9e80327be34168a9ca0af3ed81e383]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:08.980333 2482 tablet_service.cc:1467] Processing CreateTablet for tablet 2260834807694f68b5b3f079e647e43b (DEFAULT_TABLE table=ksck-test-table [id=8c9e80327be34168a9ca0af3ed81e383]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:08.981983 2482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2260834807694f68b5b3f079e647e43b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:08.985858 2480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a09692529c094da9bc48dab627698e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:08.976676 2481 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6cf7bdd47644f29fe15f5bddd380a7 (DEFAULT_TABLE table=ksck-test-table [id=8c9e80327be34168a9ca0af3ed81e383]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:08.989338 2481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6cf7bdd47644f29fe15f5bddd380a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:08.995980 2557 tablet_service.cc:1467] Processing CreateTablet for tablet a09692529c094da9bc48dab627698e3b (DEFAULT_TABLE table=ksck-test-table [id=8c9e80327be34168a9ca0af3ed81e383]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:08.998847 2557 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a09692529c094da9bc48dab627698e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:09.000931 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6cf7bdd47644f29fe15f5bddd380a7 (DEFAULT_TABLE table=ksck-test-table [id=8c9e80327be34168a9ca0af3ed81e383]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:09.012303 2559 tablet_service.cc:1467] Processing CreateTablet for tablet 2260834807694f68b5b3f079e647e43b (DEFAULT_TABLE table=ksck-test-table [id=8c9e80327be34168a9ca0af3ed81e383]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:09.005405 2558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6cf7bdd47644f29fe15f5bddd380a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:09.014719 2559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2260834807694f68b5b3f079e647e43b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:09.088557 2700 tablet_bootstrap.cc:492] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969: Bootstrap starting.
I20250115 02:52:09.109930 2701 tablet_bootstrap.cc:492] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6: Bootstrap starting.
I20250115 02:52:09.110184 2700 tablet_bootstrap.cc:654] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:09.209334 2701 tablet_bootstrap.cc:654] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:09.213433 2702 tablet_bootstrap.cc:492] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454: Bootstrap starting.
I20250115 02:52:09.225530 2700 tablet_bootstrap.cc:492] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969: No bootstrap required, opened a new log
I20250115 02:52:09.226244 2700 ts_tablet_manager.cc:1397] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969: Time spent bootstrapping tablet: real 0.139s user 0.036s sys 0.055s
I20250115 02:52:09.230276 2700 raft_consensus.cc:357] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.230841 2702 tablet_bootstrap.cc:654] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:09.231350 2700 raft_consensus.cc:383] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:09.231804 2700 raft_consensus.cc:738] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5daaa634d45349a9a9934cfa5fc76969, State: Initialized, Role: FOLLOWER
I20250115 02:52:09.241560 2700 consensus_queue.cc:260] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.252454 2702 tablet_bootstrap.cc:492] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454: No bootstrap required, opened a new log
I20250115 02:52:09.256417 2701 tablet_bootstrap.cc:492] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6: No bootstrap required, opened a new log
I20250115 02:52:09.257018 2701 ts_tablet_manager.cc:1397] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6: Time spent bootstrapping tablet: real 0.148s user 0.008s sys 0.014s
I20250115 02:52:09.261407 2701 raft_consensus.cc:357] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.262383 2701 raft_consensus.cc:383] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:09.262777 2701 raft_consensus.cc:738] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee167c4ec6e1496389be5398e565c5a6, State: Initialized, Role: FOLLOWER
I20250115 02:52:09.256495 2702 ts_tablet_manager.cc:1397] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.011s
I20250115 02:52:09.263604 2701 consensus_queue.cc:260] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.266851 2702 raft_consensus.cc:357] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.267606 2702 raft_consensus.cc:383] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:09.267874 2702 raft_consensus.cc:738] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dcb964e455f44da88719842b398c454, State: Initialized, Role: FOLLOWER
I20250115 02:52:09.268580 2702 consensus_queue.cc:260] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.273098 2700 ts_tablet_manager.cc:1428] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969: Time spent starting tablet: real 0.046s user 0.006s sys 0.016s
I20250115 02:52:09.280175 2701 ts_tablet_manager.cc:1428] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6: Time spent starting tablet: real 0.023s user 0.009s sys 0.000s
I20250115 02:52:09.281693 2702 ts_tablet_manager.cc:1428] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454: Time spent starting tablet: real 0.017s user 0.004s sys 0.006s
I20250115 02:52:09.283423 2702 tablet_bootstrap.cc:492] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454: Bootstrap starting.
I20250115 02:52:09.281798 2701 tablet_bootstrap.cc:492] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6: Bootstrap starting.
I20250115 02:52:09.294366 2701 tablet_bootstrap.cc:654] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:09.294989 2700 tablet_bootstrap.cc:492] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969: Bootstrap starting.
I20250115 02:52:09.302008 2700 tablet_bootstrap.cc:654] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:09.304301 2702 tablet_bootstrap.cc:654] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:09.306313 2701 tablet_bootstrap.cc:492] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6: No bootstrap required, opened a new log
I20250115 02:52:09.312255 2701 ts_tablet_manager.cc:1397] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.007s
I20250115 02:52:09.319352 2701 raft_consensus.cc:357] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } }
I20250115 02:52:09.321099 2701 raft_consensus.cc:383] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:09.321497 2701 raft_consensus.cc:738] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee167c4ec6e1496389be5398e565c5a6, State: Initialized, Role: FOLLOWER
I20250115 02:52:09.322374 2701 consensus_queue.cc:260] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } }
I20250115 02:52:09.327881 2700 tablet_bootstrap.cc:492] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969: No bootstrap required, opened a new log
I20250115 02:52:09.329640 2700 ts_tablet_manager.cc:1397] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.007s
I20250115 02:52:09.332159 2701 ts_tablet_manager.cc:1428] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250115 02:52:09.332893 2702 tablet_bootstrap.cc:492] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454: No bootstrap required, opened a new log
I20250115 02:52:09.333534 2702 ts_tablet_manager.cc:1397] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.003s
I20250115 02:52:09.334996 2700 raft_consensus.cc:357] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.337263 2700 raft_consensus.cc:383] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:09.336820 2702 raft_consensus.cc:357] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } }
I20250115 02:52:09.337623 2700 raft_consensus.cc:738] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5daaa634d45349a9a9934cfa5fc76969, State: Initialized, Role: FOLLOWER
I20250115 02:52:09.338037 2702 raft_consensus.cc:383] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:09.338511 2702 raft_consensus.cc:738] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dcb964e455f44da88719842b398c454, State: Initialized, Role: FOLLOWER
I20250115 02:52:09.339190 2700 consensus_queue.cc:260] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.339354 2702 consensus_queue.cc:260] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } }
I20250115 02:52:09.342149 2700 ts_tablet_manager.cc:1428] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250115 02:52:09.343856 2700 tablet_bootstrap.cc:492] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969: Bootstrap starting.
I20250115 02:52:09.343981 2702 ts_tablet_manager.cc:1428] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20250115 02:52:09.346563 2702 tablet_bootstrap.cc:492] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454: Bootstrap starting.
I20250115 02:52:09.342175 2701 tablet_bootstrap.cc:492] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6: Bootstrap starting.
I20250115 02:52:09.359548 2701 tablet_bootstrap.cc:654] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:09.361646 2702 tablet_bootstrap.cc:654] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:09.371268 2700 tablet_bootstrap.cc:654] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:09.388614 2702 tablet_bootstrap.cc:492] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454: No bootstrap required, opened a new log
I20250115 02:52:09.389119 2702 ts_tablet_manager.cc:1397] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.005s
I20250115 02:52:09.392441 2702 raft_consensus.cc:357] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.393306 2702 raft_consensus.cc:383] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:09.393579 2701 tablet_bootstrap.cc:492] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6: No bootstrap required, opened a new log
I20250115 02:52:09.393663 2702 raft_consensus.cc:738] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dcb964e455f44da88719842b398c454, State: Initialized, Role: FOLLOWER
I20250115 02:52:09.394181 2701 ts_tablet_manager.cc:1397] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.000s
I20250115 02:52:09.396849 2702 consensus_queue.cc:260] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.400631 2702 ts_tablet_manager.cc:1428] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20250115 02:52:09.403270 2700 tablet_bootstrap.cc:492] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969: No bootstrap required, opened a new log
I20250115 02:52:09.403865 2700 ts_tablet_manager.cc:1397] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969: Time spent bootstrapping tablet: real 0.060s user 0.016s sys 0.006s
I20250115 02:52:09.399451 2701 raft_consensus.cc:357] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.404489 2701 raft_consensus.cc:383] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:09.404906 2701 raft_consensus.cc:738] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee167c4ec6e1496389be5398e565c5a6, State: Initialized, Role: FOLLOWER
I20250115 02:52:09.405694 2701 consensus_queue.cc:260] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.409446 2701 ts_tablet_manager.cc:1428] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250115 02:52:09.419974 2707 raft_consensus.cc:491] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:09.420513 2707 raft_consensus.cc:513] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.427680 2707 leader_election.cc:290] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7dcb964e455f44da88719842b398c454 (127.2.36.66:43203), 5daaa634d45349a9a9934cfa5fc76969 (127.2.36.67:44463)
I20250115 02:52:09.434046 2707 raft_consensus.cc:491] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:09.435678 2707 raft_consensus.cc:513] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.442626 2707 leader_election.cc:290] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7dcb964e455f44da88719842b398c454 (127.2.36.66:43203), 5daaa634d45349a9a9934cfa5fc76969 (127.2.36.67:44463)
I20250115 02:52:09.446108 2700 raft_consensus.cc:357] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } }
I20250115 02:52:09.447033 2700 raft_consensus.cc:383] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:09.447382 2700 raft_consensus.cc:738] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5daaa634d45349a9a9934cfa5fc76969, State: Initialized, Role: FOLLOWER
I20250115 02:52:09.448179 2700 consensus_queue.cc:260] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } }
I20250115 02:52:09.452471 2700 ts_tablet_manager.cc:1428] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969: Time spent starting tablet: real 0.048s user 0.005s sys 0.000s
I20250115 02:52:09.461789 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6cf7bdd47644f29fe15f5bddd380a7" candidate_uuid: "ee167c4ec6e1496389be5398e565c5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5daaa634d45349a9a9934cfa5fc76969" is_pre_election: true
I20250115 02:52:09.462170 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6cf7bdd47644f29fe15f5bddd380a7" candidate_uuid: "ee167c4ec6e1496389be5398e565c5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcb964e455f44da88719842b398c454" is_pre_election: true
I20250115 02:52:09.462954 2706 raft_consensus.cc:491] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:09.462744 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09692529c094da9bc48dab627698e3b" candidate_uuid: "ee167c4ec6e1496389be5398e565c5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5daaa634d45349a9a9934cfa5fc76969" is_pre_election: true
I20250115 02:52:09.463490 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09692529c094da9bc48dab627698e3b" candidate_uuid: "ee167c4ec6e1496389be5398e565c5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcb964e455f44da88719842b398c454" is_pre_election: true
I20250115 02:52:09.463658 2706 raft_consensus.cc:513] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.464254 2565 raft_consensus.cc:2463] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee167c4ec6e1496389be5398e565c5a6 in term 0.
I20250115 02:52:09.464275 2646 raft_consensus.cc:2463] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee167c4ec6e1496389be5398e565c5a6 in term 0.
I20250115 02:52:09.466203 2457 leader_election.cc:304] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5daaa634d45349a9a9934cfa5fc76969, ee167c4ec6e1496389be5398e565c5a6; no voters:
I20250115 02:52:09.468519 2458 leader_election.cc:304] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dcb964e455f44da88719842b398c454, ee167c4ec6e1496389be5398e565c5a6; no voters:
I20250115 02:52:09.463035 2569 raft_consensus.cc:2463] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee167c4ec6e1496389be5398e565c5a6 in term 0.
I20250115 02:52:09.470443 2706 leader_election.cc:290] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee167c4ec6e1496389be5398e565c5a6 (127.2.36.65:44597), 7dcb964e455f44da88719842b398c454 (127.2.36.66:43203)
I20250115 02:52:09.470929 2707 raft_consensus.cc:2798] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:09.478488 2707 raft_consensus.cc:491] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:09.479033 2707 raft_consensus.cc:3054] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:09.480100 2708 raft_consensus.cc:491] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:09.480563 2708 raft_consensus.cc:513] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } }
I20250115 02:52:09.471587 2645 raft_consensus.cc:2463] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee167c4ec6e1496389be5398e565c5a6 in term 0.
I20250115 02:52:09.489495 2708 leader_election.cc:290] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5daaa634d45349a9a9934cfa5fc76969 (127.2.36.67:44463), ee167c4ec6e1496389be5398e565c5a6 (127.2.36.65:44597)
I20250115 02:52:09.504674 2707 raft_consensus.cc:513] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.506670 2707 leader_election.cc:290] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [CANDIDATE]: Term 1 election: Requested vote from peers 7dcb964e455f44da88719842b398c454 (127.2.36.66:43203), 5daaa634d45349a9a9934cfa5fc76969 (127.2.36.67:44463)
I20250115 02:52:09.518437 2707 raft_consensus.cc:491] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:09.518946 2707 raft_consensus.cc:513] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } }
I20250115 02:52:09.521466 2707 leader_election.cc:290] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5daaa634d45349a9a9934cfa5fc76969 (127.2.36.67:44463), 7dcb964e455f44da88719842b398c454 (127.2.36.66:43203)
I20250115 02:52:09.522073 2707 raft_consensus.cc:2798] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:09.522480 2707 raft_consensus.cc:491] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:09.522934 2707 raft_consensus.cc:3054] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Advancing to term 1
W20250115 02:52:09.516539 2458 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.36.66:43203, user_credentials={real_user=slave}} blocked reactor thread for 48179.4us
I20250115 02:52:09.535826 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6cf7bdd47644f29fe15f5bddd380a7" candidate_uuid: "ee167c4ec6e1496389be5398e565c5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcb964e455f44da88719842b398c454"
I20250115 02:52:09.534827 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6cf7bdd47644f29fe15f5bddd380a7" candidate_uuid: "ee167c4ec6e1496389be5398e565c5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5daaa634d45349a9a9934cfa5fc76969"
I20250115 02:52:09.536624 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2260834807694f68b5b3f079e647e43b" candidate_uuid: "ee167c4ec6e1496389be5398e565c5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcb964e455f44da88719842b398c454" is_pre_election: true
I20250115 02:52:09.536859 2645 raft_consensus.cc:3054] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:09.537289 2565 raft_consensus.cc:2463] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee167c4ec6e1496389be5398e565c5a6 in term 0.
I20250115 02:52:09.539932 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2260834807694f68b5b3f079e647e43b" candidate_uuid: "ee167c4ec6e1496389be5398e565c5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5daaa634d45349a9a9934cfa5fc76969" is_pre_election: true
I20250115 02:52:09.541038 2646 raft_consensus.cc:2463] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee167c4ec6e1496389be5398e565c5a6 in term 0.
I20250115 02:52:09.539448 2569 raft_consensus.cc:3054] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:09.552356 2457 leader_election.cc:304] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5daaa634d45349a9a9934cfa5fc76969, ee167c4ec6e1496389be5398e565c5a6; no voters:
I20250115 02:52:09.558678 2707 raft_consensus.cc:513] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.559170 2645 raft_consensus.cc:2463] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee167c4ec6e1496389be5398e565c5a6 in term 1.
I20250115 02:52:09.560344 2719 raft_consensus.cc:2798] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:09.560825 2719 raft_consensus.cc:491] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:09.561336 2719 raft_consensus.cc:3054] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:09.562836 2707 leader_election.cc:290] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [CANDIDATE]: Term 1 election: Requested vote from peers 7dcb964e455f44da88719842b398c454 (127.2.36.66:43203), 5daaa634d45349a9a9934cfa5fc76969 (127.2.36.67:44463)
I20250115 02:52:09.566208 2569 raft_consensus.cc:2463] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee167c4ec6e1496389be5398e565c5a6 in term 1.
I20250115 02:52:09.574998 2458 leader_election.cc:304] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dcb964e455f44da88719842b398c454, ee167c4ec6e1496389be5398e565c5a6; no voters:
I20250115 02:52:09.575374 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09692529c094da9bc48dab627698e3b" candidate_uuid: "ee167c4ec6e1496389be5398e565c5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcb964e455f44da88719842b398c454"
I20250115 02:52:09.576162 2569 raft_consensus.cc:3054] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:09.577363 2707 raft_consensus.cc:2798] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:09.582392 2707 raft_consensus.cc:695] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 1 LEADER]: Becoming Leader. State: Replica: ee167c4ec6e1496389be5398e565c5a6, State: Running, Role: LEADER
I20250115 02:52:09.583338 2707 consensus_queue.cc:237] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.585798 2569 raft_consensus.cc:2463] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee167c4ec6e1496389be5398e565c5a6 in term 1.
I20250115 02:52:09.587276 2458 leader_election.cc:304] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dcb964e455f44da88719842b398c454, ee167c4ec6e1496389be5398e565c5a6; no voters:
I20250115 02:52:09.590950 2720 raft_consensus.cc:2798] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:09.599318 2720 raft_consensus.cc:695] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 1 LEADER]: Becoming Leader. State: Replica: ee167c4ec6e1496389be5398e565c5a6, State: Running, Role: LEADER
I20250115 02:52:09.611119 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09692529c094da9bc48dab627698e3b" candidate_uuid: "5daaa634d45349a9a9934cfa5fc76969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcb964e455f44da88719842b398c454" is_pre_election: true
I20250115 02:52:09.613054 2569 raft_consensus.cc:2388] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5daaa634d45349a9a9934cfa5fc76969 in current term 1: Already voted for candidate ee167c4ec6e1496389be5398e565c5a6 in this term.
I20250115 02:52:09.621069 2720 consensus_queue.cc:237] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } }
I20250115 02:52:09.661696 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09692529c094da9bc48dab627698e3b" candidate_uuid: "ee167c4ec6e1496389be5398e565c5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5daaa634d45349a9a9934cfa5fc76969"
I20250115 02:52:09.666575 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2260834807694f68b5b3f079e647e43b" candidate_uuid: "7dcb964e455f44da88719842b398c454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5daaa634d45349a9a9934cfa5fc76969" is_pre_election: true
I20250115 02:52:09.663671 2645 raft_consensus.cc:3054] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:09.665827 2719 raft_consensus.cc:513] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } }
I20250115 02:52:09.667309 2646 raft_consensus.cc:2463] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dcb964e455f44da88719842b398c454 in term 0.
I20250115 02:52:09.680835 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2260834807694f68b5b3f079e647e43b" candidate_uuid: "ee167c4ec6e1496389be5398e565c5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcb964e455f44da88719842b398c454"
I20250115 02:52:09.681529 2569 raft_consensus.cc:3054] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:09.681789 2645 raft_consensus.cc:2463] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee167c4ec6e1496389be5398e565c5a6 in term 1.
I20250115 02:52:09.680688 2719 leader_election.cc:290] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [CANDIDATE]: Term 1 election: Requested vote from peers 5daaa634d45349a9a9934cfa5fc76969 (127.2.36.67:44463), 7dcb964e455f44da88719842b398c454 (127.2.36.66:43203)
I20250115 02:52:09.683256 2534 leader_election.cc:304] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5daaa634d45349a9a9934cfa5fc76969, 7dcb964e455f44da88719842b398c454; no voters:
I20250115 02:52:09.688023 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2260834807694f68b5b3f079e647e43b" candidate_uuid: "ee167c4ec6e1496389be5398e565c5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5daaa634d45349a9a9934cfa5fc76969"
I20250115 02:52:09.689548 2645 raft_consensus.cc:3054] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:09.691560 2569 raft_consensus.cc:2463] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee167c4ec6e1496389be5398e565c5a6 in term 1.
I20250115 02:52:09.691972 2492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2260834807694f68b5b3f079e647e43b" candidate_uuid: "7dcb964e455f44da88719842b398c454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee167c4ec6e1496389be5398e565c5a6" is_pre_election: true
I20250115 02:52:09.692381 2491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09692529c094da9bc48dab627698e3b" candidate_uuid: "5daaa634d45349a9a9934cfa5fc76969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee167c4ec6e1496389be5398e565c5a6" is_pre_election: true
I20250115 02:52:09.692974 2492 raft_consensus.cc:2388] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7dcb964e455f44da88719842b398c454 in current term 1: Already voted for candidate ee167c4ec6e1496389be5398e565c5a6 in this term.
I20250115 02:52:09.694931 2708 raft_consensus.cc:2758] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250115 02:52:09.696247 2458 leader_election.cc:304] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dcb964e455f44da88719842b398c454, ee167c4ec6e1496389be5398e565c5a6; no voters:
I20250115 02:52:09.697234 2720 raft_consensus.cc:2798] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:09.699411 2720 raft_consensus.cc:695] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 1 LEADER]: Becoming Leader. State: Replica: ee167c4ec6e1496389be5398e565c5a6, State: Running, Role: LEADER
I20250115 02:52:09.703320 2720 consensus_queue.cc:237] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } } peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } }
I20250115 02:52:09.704597 2355 catalog_manager.cc:5526] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee167c4ec6e1496389be5398e565c5a6 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "ee167c4ec6e1496389be5398e565c5a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:09.707113 2355 catalog_manager.cc:5526] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee167c4ec6e1496389be5398e565c5a6 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "ee167c4ec6e1496389be5398e565c5a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:09.708648 2645 raft_consensus.cc:2463] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee167c4ec6e1496389be5398e565c5a6 in term 1.
I20250115 02:52:09.711011 2611 leader_election.cc:304] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5daaa634d45349a9a9934cfa5fc76969; no voters: 7dcb964e455f44da88719842b398c454, ee167c4ec6e1496389be5398e565c5a6
I20250115 02:52:09.712193 2706 raft_consensus.cc:2743] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:52:09.785257 2355 catalog_manager.cc:5526] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee167c4ec6e1496389be5398e565c5a6 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "ee167c4ec6e1496389be5398e565c5a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee167c4ec6e1496389be5398e565c5a6" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44597 } health_report { overall_health: HEALTHY } } }
I20250115 02:52:10.021027 2720 consensus_queue.cc:1035] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:10.051916 2720 consensus_queue.cc:1035] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250115 02:52:10.063671 2724 consensus_queue.cc:1035] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:10.072371 2719 consensus_queue.cc:1035] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:10.222534 2724 consensus_queue.cc:1035] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5daaa634d45349a9a9934cfa5fc76969" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:10.270148 2724 consensus_queue.cc:1035] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dcb964e455f44da88719842b398c454" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:11.201828 2557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:52:11.211869 2634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:52:11.244499 2481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250115 02:52:11.586575 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:52:11.630875 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:11.631438 2193 tablet_replica.cc:331] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6: stopping tablet replica
I20250115 02:52:11.632092 2193 raft_consensus.cc:2238] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:11.633038 2193 raft_consensus.cc:2267] T 3b6cf7bdd47644f29fe15f5bddd380a7 P ee167c4ec6e1496389be5398e565c5a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:11.636263 2193 tablet_replica.cc:331] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6: stopping tablet replica
I20250115 02:52:11.636731 2193 raft_consensus.cc:2238] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:11.637580 2193 raft_consensus.cc:2267] T 2260834807694f68b5b3f079e647e43b P ee167c4ec6e1496389be5398e565c5a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:11.639361 2193 tablet_replica.cc:331] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6: stopping tablet replica
I20250115 02:52:11.640003 2193 raft_consensus.cc:2238] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:11.640650 2193 raft_consensus.cc:2267] T a09692529c094da9bc48dab627698e3b P ee167c4ec6e1496389be5398e565c5a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:11.817864 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:52:11.831856 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:52:11.864809 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:11.865607 2193 tablet_replica.cc:331] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454: stopping tablet replica
I20250115 02:52:11.866277 2193 raft_consensus.cc:2238] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:11.866673 2193 raft_consensus.cc:2267] T a09692529c094da9bc48dab627698e3b P 7dcb964e455f44da88719842b398c454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:11.868459 2193 tablet_replica.cc:331] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454: stopping tablet replica
I20250115 02:52:11.868894 2193 raft_consensus.cc:2238] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:11.869275 2193 raft_consensus.cc:2267] T 2260834807694f68b5b3f079e647e43b P 7dcb964e455f44da88719842b398c454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:11.870731 2193 tablet_replica.cc:331] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454: stopping tablet replica
I20250115 02:52:11.871196 2193 raft_consensus.cc:2238] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:11.871554 2193 raft_consensus.cc:2267] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 7dcb964e455f44da88719842b398c454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:11.892390 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:52:11.902566 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:52:11.926116 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:11.926640 2193 tablet_replica.cc:331] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969: stopping tablet replica
I20250115 02:52:11.927198 2193 raft_consensus.cc:2238] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:11.927595 2193 raft_consensus.cc:2267] T 2260834807694f68b5b3f079e647e43b P 5daaa634d45349a9a9934cfa5fc76969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:11.929068 2193 tablet_replica.cc:331] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969: stopping tablet replica
I20250115 02:52:11.929507 2193 raft_consensus.cc:2238] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:11.929852 2193 raft_consensus.cc:2267] T 3b6cf7bdd47644f29fe15f5bddd380a7 P 5daaa634d45349a9a9934cfa5fc76969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:11.931288 2193 tablet_replica.cc:331] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969: stopping tablet replica
I20250115 02:52:11.931694 2193 raft_consensus.cc:2238] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:11.932024 2193 raft_consensus.cc:2267] T a09692529c094da9bc48dab627698e3b P 5daaa634d45349a9a9934cfa5fc76969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:11.950610 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:52:11.959937 2193 master.cc:537] Master@127.2.36.126:45827 shutting down...
I20250115 02:52:12.063827 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:12.064323 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:12.064635 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P c977b7e3ed80457ab586c6445e72da06: stopping tablet replica
I20250115 02:52:12.082962 2193 master.cc:559] Master@127.2.36.126:45827 shutdown complete.
I20250115 02:52:12.094447 2193 master.cc:537] Master@127.2.36.125:42201 shutting down...
I20250115 02:52:12.108738 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:12.109215 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:12.109498 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P cb1f33e2785e418b8324b1306d39a562: stopping tablet replica
I20250115 02:52:12.129822 2193 master.cc:559] Master@127.2.36.125:42201 shutdown complete.
I20250115 02:52:12.138784 2193 master.cc:537] Master@127.2.36.124:45025 shutting down...
I20250115 02:52:12.153885 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:12.154657 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:12.155057 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7749ab69bfeb401b89687b764189ed9a: stopping tablet replica
I20250115 02:52:12.173309 2193 master.cc:559] Master@127.2.36.124:45025 shutdown complete.
[ OK ] RemoteKsckTest.TestClusterOk (6240 ms)
[ RUN ] RemoteKsckTest.TestCheckUnusualFlags
I20250115 02:52:12.200929 2193 test_util.cc:274] Using random seed: -1012105268
I20250115 02:52:12.201875 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:44129,127.2.36.125:40933,127.2.36.124:36933
I20250115 02:52:12.203166 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:12.208576 2786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:12.209200 2787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:12.210151 2789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:12.211079 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:12.211771 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:12.211954 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:12.212090 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909532212074 us; error 0 us; skew 500 ppm
I20250115 02:52:12.212607 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:12.218274 2193 webserver.cc:458] Webserver started at http://127.2.36.126:33781/ using document root <none> and password file <none>
I20250115 02:52:12.218740 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:12.218955 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:12.219184 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:12.220175 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "f79449208ad84e708353086b800a3b9a"
format_stamp: "Formatted at 2025-01-15 02:52:12 on dist-test-slave-1lv2"
I20250115 02:52:12.224249 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:52:12.227273 2794 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:12.227955 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:52:12.228221 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "f79449208ad84e708353086b800a3b9a"
format_stamp: "Formatted at 2025-01-15 02:52:12 on dist-test-slave-1lv2"
I20250115 02:52:12.228471 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:12.248145 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:12.249121 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:12.282167 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:44129
I20250115 02:52:12.282250 2845 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:44129 every 8 connection(s)
I20250115 02:52:12.285890 2846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:12.286163 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:12.292317 2848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:12.292457 2846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:12.293236 2849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:12.294075 2851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:12.297448 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:12.298261 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:12.298532 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:12.298734 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909532298715 us; error 0 us; skew 500 ppm
I20250115 02:52:12.299425 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:12.302064 2193 webserver.cc:458] Webserver started at http://127.2.36.125:45431/ using document root <none> and password file <none>
I20250115 02:52:12.302610 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:12.302835 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:12.303148 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:12.304481 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "6cf4710fe9f9489eb610430ea9fc77d9"
format_stamp: "Formatted at 2025-01-15 02:52:12 on dist-test-slave-1lv2"
I20250115 02:52:12.305294 2846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:12.309790 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
W20250115 02:52:12.310560 2846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:40933: Network error: Client connection negotiation failed: client connection to 127.2.36.125:40933: connect: Connection refused (error 111)
I20250115 02:52:12.313164 2859 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:12.313889 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:52:12.314167 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "6cf4710fe9f9489eb610430ea9fc77d9"
format_stamp: "Formatted at 2025-01-15 02:52:12 on dist-test-slave-1lv2"
I20250115 02:52:12.314441 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:12.325573 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:12.326516 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:12.341027 2846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } attempt: 1
W20250115 02:52:12.343679 2797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.36.125:40933: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20250115 02:52:12.346544 2846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:40933: Network error: Client connection negotiation failed: client connection to 127.2.36.125:40933: connect: Connection refused (error 111)
I20250115 02:52:12.360644 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:40933
I20250115 02:52:12.360714 2910 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:40933 every 8 connection(s)
I20250115 02:52:12.364225 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:12.364468 2911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 02:52:12.370188 2913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:12.370812 2914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:12.372071 2911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:12.373085 2916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:12.374207 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:12.375309 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:12.375581 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:12.375782 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909532375763 us; error 0 us; skew 500 ppm
I20250115 02:52:12.376363 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:12.378899 2193 webserver.cc:458] Webserver started at http://127.2.36.124:39405/ using document root <none> and password file <none>
I20250115 02:52:12.379487 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:12.379716 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:12.380014 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:12.381327 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "98d67f98112b4b1d9d1a849c730c3473"
format_stamp: "Formatted at 2025-01-15 02:52:12 on dist-test-slave-1lv2"
I20250115 02:52:12.383761 2911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:12.386844 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20250115 02:52:12.390236 2923 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:12.390069 2846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } attempt: 2
I20250115 02:52:12.391372 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250115 02:52:12.391687 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "98d67f98112b4b1d9d1a849c730c3473"
format_stamp: "Formatted at 2025-01-15 02:52:12 on dist-test-slave-1lv2"
I20250115 02:52:12.392055 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:12.395013 2911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:12.401365 2911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:36933: Network error: Client connection negotiation failed: client connection to 127.2.36.124:36933: connect: Connection refused (error 111)
I20250115 02:52:12.403664 2846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:12.407639 2846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:36933: Network error: Client connection negotiation failed: client connection to 127.2.36.124:36933: connect: Connection refused (error 111)
I20250115 02:52:12.413964 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:12.415031 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:12.430846 2911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } attempt: 1
W20250115 02:52:12.435559 2911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:36933: Network error: Client connection negotiation failed: client connection to 127.2.36.124:36933: connect: Connection refused (error 111)
I20250115 02:52:12.450170 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:36933
I20250115 02:52:12.450249 2976 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:36933 every 8 connection(s)
I20250115 02:52:12.453061 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:52:12.453974 2977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:12.459290 2977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:12.469338 2977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:12.470098 2846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } attempt: 1
I20250115 02:52:12.473985 2911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } attempt: 2
I20250115 02:52:12.482023 2977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:12.488602 2846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a: Bootstrap starting.
I20250115 02:52:12.492570 2911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9: Bootstrap starting.
I20250115 02:52:12.494293 2846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:12.498205 2846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a: No bootstrap required, opened a new log
I20250115 02:52:12.498466 2911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:12.500164 2846 raft_consensus.cc:357] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } }
I20250115 02:52:12.500757 2846 raft_consensus.cc:383] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:12.500834 2977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473: Bootstrap starting.
I20250115 02:52:12.501096 2846 raft_consensus.cc:738] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79449208ad84e708353086b800a3b9a, State: Initialized, Role: FOLLOWER
I20250115 02:52:12.501885 2846 consensus_queue.cc:260] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } }
I20250115 02:52:12.503540 2911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9: No bootstrap required, opened a new log
I20250115 02:52:12.503908 2984 sys_catalog.cc:455] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } } }
I20250115 02:52:12.504778 2984 sys_catalog.cc:458] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:12.505443 2846 sys_catalog.cc:564] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:12.506894 2977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:12.506471 2911 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } }
I20250115 02:52:12.507282 2911 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:12.507617 2911 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf4710fe9f9489eb610430ea9fc77d9, State: Initialized, Role: FOLLOWER
I20250115 02:52:12.508499 2911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } }
I20250115 02:52:12.510994 2989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } } }
I20250115 02:52:12.511940 2989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:12.512719 2911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:12.515584 2977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473: No bootstrap required, opened a new log
I20250115 02:52:12.518543 2977 raft_consensus.cc:357] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } }
I20250115 02:52:12.519376 2977 raft_consensus.cc:383] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:12.519738 2977 raft_consensus.cc:738] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d67f98112b4b1d9d1a849c730c3473, State: Initialized, Role: FOLLOWER
I20250115 02:52:12.520546 2977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } }
I20250115 02:52:12.522934 3001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } } }
I20250115 02:52:12.524374 3001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:12.524660 2977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:12.525874 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20250115 02:52:12.529044 3003 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:12.529320 3003 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:12.534024 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250115 02:52:12.537165 3014 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:12.537439 3014 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:12.540925 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:12.543622 3020 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:12.543985 3020 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:52:12.548758 3021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:12.549350 3022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:12.551010 3024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:12.551667 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:12.552369 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:12.552590 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:12.552742 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909532552728 us; error 0 us; skew 500 ppm
I20250115 02:52:12.553228 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:12.555565 2193 webserver.cc:458] Webserver started at http://127.2.36.65:39005/ using document root <none> and password file <none>
I20250115 02:52:12.556068 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:12.556253 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:12.556499 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:12.557502 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "e0f47f62dbab4a96a62923bb11eec90b"
format_stamp: "Formatted at 2025-01-15 02:52:12 on dist-test-slave-1lv2"
I20250115 02:52:12.561643 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250115 02:52:12.564694 3029 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:12.565441 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:52:12.565721 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "e0f47f62dbab4a96a62923bb11eec90b"
format_stamp: "Formatted at 2025-01-15 02:52:12 on dist-test-slave-1lv2"
I20250115 02:52:12.565999 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:12.574990 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:12.575924 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:12.577209 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:12.579533 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:12.579782 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:12.579988 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:12.580109 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:12.615463 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:37049
I20250115 02:52:12.615569 3091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:37049 every 8 connection(s)
I20250115 02:52:12.622545 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:12.635440 3094 heartbeater.cc:346] Connected to a master server at 127.2.36.126:44129
I20250115 02:52:12.635926 3094 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:12.636828 3092 heartbeater.cc:346] Connected to a master server at 127.2.36.124:36933
I20250115 02:52:12.637445 3092 heartbeater.cc:463] Registering TS with master...
W20250115 02:52:12.638104 3099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:12.638491 3092 heartbeater.cc:509] Master 127.2.36.124:36933 requested a full tablet report, sending...
I20250115 02:52:12.636880 3094 heartbeater.cc:509] Master 127.2.36.126:44129 requested a full tablet report, sending...
W20250115 02:52:12.641157 3100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:12.641891 2942 ts_manager.cc:194] Registered new tserver with Master: e0f47f62dbab4a96a62923bb11eec90b (127.2.36.65:37049)
I20250115 02:52:12.642124 2811 ts_manager.cc:194] Registered new tserver with Master: e0f47f62dbab4a96a62923bb11eec90b (127.2.36.65:37049)
W20250115 02:52:12.646704 3102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:12.647606 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:12.648602 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:12.648802 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:12.648995 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909532648977 us; error 0 us; skew 500 ppm
I20250115 02:52:12.649096 3093 heartbeater.cc:346] Connected to a master server at 127.2.36.125:40933
I20250115 02:52:12.649484 3093 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:12.649658 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:12.650218 3093 heartbeater.cc:509] Master 127.2.36.125:40933 requested a full tablet report, sending...
I20250115 02:52:12.652484 2193 webserver.cc:458] Webserver started at http://127.2.36.66:34477/ using document root <none> and password file <none>
I20250115 02:52:12.652520 2876 ts_manager.cc:194] Registered new tserver with Master: e0f47f62dbab4a96a62923bb11eec90b (127.2.36.65:37049)
I20250115 02:52:12.653065 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:12.653286 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:12.653594 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:12.654948 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8"
format_stamp: "Formatted at 2025-01-15 02:52:12 on dist-test-slave-1lv2"
I20250115 02:52:12.660092 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250115 02:52:12.664593 3107 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:12.666702 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250115 02:52:12.667105 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8"
format_stamp: "Formatted at 2025-01-15 02:52:12 on dist-test-slave-1lv2"
I20250115 02:52:12.667490 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:12.689287 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:12.690392 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:12.691802 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:12.694243 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:12.694515 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:12.694816 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:12.694990 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:12.739439 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:38557
I20250115 02:52:12.739531 3169 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:38557 every 8 connection(s)
I20250115 02:52:12.746162 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:12.762986 3170 heartbeater.cc:346] Connected to a master server at 127.2.36.124:36933
I20250115 02:52:12.763429 3170 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:12.764312 3170 heartbeater.cc:509] Master 127.2.36.124:36933 requested a full tablet report, sending...
W20250115 02:52:12.765277 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:12.766921 2942 ts_manager.cc:194] Registered new tserver with Master: dcdee8bc1ad94b7f9db7c526a7615bd8 (127.2.36.66:38557)
I20250115 02:52:12.770445 3171 heartbeater.cc:346] Connected to a master server at 127.2.36.125:40933
I20250115 02:52:12.770859 3171 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:12.771770 3171 heartbeater.cc:509] Master 127.2.36.125:40933 requested a full tablet report, sending...
I20250115 02:52:12.774312 2876 ts_manager.cc:194] Registered new tserver with Master: dcdee8bc1ad94b7f9db7c526a7615bd8 (127.2.36.66:38557)
I20250115 02:52:12.775972 3172 heartbeater.cc:346] Connected to a master server at 127.2.36.126:44129
I20250115 02:52:12.776432 3172 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:12.777390 3172 heartbeater.cc:509] Master 127.2.36.126:44129 requested a full tablet report, sending...
W20250115 02:52:12.777470 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:12.778553 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:12.780306 2811 ts_manager.cc:194] Registered new tserver with Master: dcdee8bc1ad94b7f9db7c526a7615bd8 (127.2.36.66:38557)
I20250115 02:52:12.785871 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:12.786892 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:12.787153 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:12.787339 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909532787322 us; error 0 us; skew 500 ppm
I20250115 02:52:12.787842 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:12.790175 2193 webserver.cc:458] Webserver started at http://127.2.36.67:39143/ using document root <none> and password file <none>
I20250115 02:52:12.790635 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:12.790844 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:12.791085 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:12.792248 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "1c80c0bbf6924856bac7d2e5cb49434a"
format_stamp: "Formatted at 2025-01-15 02:52:12 on dist-test-slave-1lv2"
I20250115 02:52:12.801127 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.004s
I20250115 02:52:12.804370 3185 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:12.805111 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250115 02:52:12.805423 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "1c80c0bbf6924856bac7d2e5cb49434a"
format_stamp: "Formatted at 2025-01-15 02:52:12 on dist-test-slave-1lv2"
I20250115 02:52:12.805687 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestCheckUnusualFlags.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:12.824151 2989 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:12.824682 2989 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } }
I20250115 02:52:12.825058 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:12.826632 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:12.826717 2989 leader_election.cc:290] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f79449208ad84e708353086b800a3b9a (127.2.36.126:44129), 98d67f98112b4b1d9d1a849c730c3473 (127.2.36.124:36933)
I20250115 02:52:12.827934 2952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d67f98112b4b1d9d1a849c730c3473" is_pre_election: true
I20250115 02:52:12.831298 2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79449208ad84e708353086b800a3b9a" is_pre_election: true
I20250115 02:52:12.831585 2952 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf4710fe9f9489eb610430ea9fc77d9 in term 0.
I20250115 02:52:12.832018 2821 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf4710fe9f9489eb610430ea9fc77d9 in term 0.
I20250115 02:52:12.833052 2862 leader_election.cc:304] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf4710fe9f9489eb610430ea9fc77d9, 98d67f98112b4b1d9d1a849c730c3473; no voters:
I20250115 02:52:12.834228 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:12.835690 2989 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:12.836198 2989 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:12.836640 2989 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:12.839244 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:12.839565 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:12.839865 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:12.840076 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:12.843212 2989 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } }
I20250115 02:52:12.845222 2989 leader_election.cc:290] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [CANDIDATE]: Term 1 election: Requested vote from peers f79449208ad84e708353086b800a3b9a (127.2.36.126:44129), 98d67f98112b4b1d9d1a849c730c3473 (127.2.36.124:36933)
I20250115 02:52:12.846377 2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79449208ad84e708353086b800a3b9a"
I20250115 02:52:12.846783 2952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d67f98112b4b1d9d1a849c730c3473"
I20250115 02:52:12.847390 2952 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:12.847837 2821 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:12.855012 2821 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf4710fe9f9489eb610430ea9fc77d9 in term 1.
I20250115 02:52:12.857798 2952 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf4710fe9f9489eb610430ea9fc77d9 in term 1.
I20250115 02:52:12.857736 2863 leader_election.cc:304] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf4710fe9f9489eb610430ea9fc77d9, f79449208ad84e708353086b800a3b9a; no voters:
I20250115 02:52:12.859064 2989 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:12.864192 2989 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 1 LEADER]: Becoming Leader. State: Replica: 6cf4710fe9f9489eb610430ea9fc77d9, State: Running, Role: LEADER
I20250115 02:52:12.865160 2989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } }
I20250115 02:52:12.875936 3196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cf4710fe9f9489eb610430ea9fc77d9. Latest consensus state: current_term: 1 leader_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } } }
I20250115 02:52:12.876775 3196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:12.880573 3205 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:52:12.891558 3205 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:52:12.956190 2952 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [term 1 FOLLOWER]: Refusing update from remote peer 6cf4710fe9f9489eb610430ea9fc77d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:12.956817 2821 raft_consensus.cc:1270] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [term 1 FOLLOWER]: Refusing update from remote peer 6cf4710fe9f9489eb610430ea9fc77d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:12.957540 3196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:12.958410 2989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:12.973558 3001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cf4710fe9f9489eb610430ea9fc77d9. Latest consensus state: current_term: 1 leader_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } } }
I20250115 02:52:12.974252 3001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:12.981485 3205 catalog_manager.cc:1348] Generated new cluster ID: 995713105d22453daa382da688b1116c
I20250115 02:52:12.981887 3205 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:52:12.980437 2989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } } }
I20250115 02:52:12.994683 2989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:13.005985 3001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } } }
I20250115 02:52:13.008105 3001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:13.016919 2984 sys_catalog.cc:455] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cf4710fe9f9489eb610430ea9fc77d9. Latest consensus state: current_term: 1 leader_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } } }
I20250115 02:52:13.017762 2984 sys_catalog.cc:458] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:13.022639 2989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } } }
I20250115 02:52:13.023555 2989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:13.035216 3205 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:52:13.037017 3205 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:52:13.049772 2984 sys_catalog.cc:455] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79449208ad84e708353086b800a3b9a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 44129 } } peers { permanent_uuid: "6cf4710fe9f9489eb610430ea9fc77d9" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40933 } } peers { permanent_uuid: "98d67f98112b4b1d9d1a849c730c3473" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36933 } } }
I20250115 02:52:13.050683 2984 sys_catalog.cc:458] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:13.079435 3205 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9: Generated new TSK 0
I20250115 02:52:13.080274 3205 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:52:13.102880 2876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57388
I20250115 02:52:13.104032 2875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39618
I20250115 02:52:13.109493 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:41755
I20250115 02:52:13.109607 3259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:41755 every 8 connection(s)
I20250115 02:52:13.117025 3171 heartbeater.cc:501] Master 127.2.36.125:40933 was elected leader, sending a full tablet report...
I20250115 02:52:13.117400 3093 heartbeater.cc:501] Master 127.2.36.125:40933 was elected leader, sending a full tablet report...
I20250115 02:52:13.135176 3260 heartbeater.cc:346] Connected to a master server at 127.2.36.124:36933
I20250115 02:52:13.137499 3260 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:13.138245 3260 heartbeater.cc:509] Master 127.2.36.124:36933 requested a full tablet report, sending...
I20250115 02:52:13.140595 3261 heartbeater.cc:346] Connected to a master server at 127.2.36.125:40933
I20250115 02:52:13.140941 3261 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:13.141525 2941 ts_manager.cc:194] Registered new tserver with Master: 1c80c0bbf6924856bac7d2e5cb49434a (127.2.36.67:41755)
I20250115 02:52:13.141791 3261 heartbeater.cc:509] Master 127.2.36.125:40933 requested a full tablet report, sending...
I20250115 02:52:13.144024 2875 ts_manager.cc:194] Registered new tserver with Master: 1c80c0bbf6924856bac7d2e5cb49434a (127.2.36.67:41755)
I20250115 02:52:13.145665 2875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57402
I20250115 02:52:13.148968 3262 heartbeater.cc:346] Connected to a master server at 127.2.36.126:44129
I20250115 02:52:13.149379 3262 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:13.149937 3262 heartbeater.cc:509] Master 127.2.36.126:44129 requested a full tablet report, sending...
I20250115 02:52:13.152323 2810 ts_manager.cc:194] Registered new tserver with Master: 1c80c0bbf6924856bac7d2e5cb49434a (127.2.36.67:41755)
I20250115 02:52:13.153134 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.0275473s
I20250115 02:52:13.158241 3261 heartbeater.cc:501] Master 127.2.36.125:40933 was elected leader, sending a full tablet report...
I20250115 02:52:13.193763 2875 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57418:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:52:13.195974 2875 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:52:13.300637 3216 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9c9757227d40b8913e577616004b6e (DEFAULT_TABLE table=ksck-test-table [id=59c4365bf0a8428f9f1084740e627a88]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:13.301990 3216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9c9757227d40b8913e577616004b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:13.303766 3215 tablet_service.cc:1467] Processing CreateTablet for tablet fcff98b6f57140d38ebb0a5b12186797 (DEFAULT_TABLE table=ksck-test-table [id=59c4365bf0a8428f9f1084740e627a88]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:13.305104 3215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcff98b6f57140d38ebb0a5b12186797. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:13.307631 3214 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4c06fe0ea746239f6462fbbdd45a4e (DEFAULT_TABLE table=ksck-test-table [id=59c4365bf0a8428f9f1084740e627a88]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:13.308738 3214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c4c06fe0ea746239f6462fbbdd45a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:13.309804 3135 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9c9757227d40b8913e577616004b6e (DEFAULT_TABLE table=ksck-test-table [id=59c4365bf0a8428f9f1084740e627a88]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:13.311077 3135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9c9757227d40b8913e577616004b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:13.311792 3134 tablet_service.cc:1467] Processing CreateTablet for tablet fcff98b6f57140d38ebb0a5b12186797 (DEFAULT_TABLE table=ksck-test-table [id=59c4365bf0a8428f9f1084740e627a88]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:13.312785 3134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcff98b6f57140d38ebb0a5b12186797. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:13.313899 3133 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4c06fe0ea746239f6462fbbdd45a4e (DEFAULT_TABLE table=ksck-test-table [id=59c4365bf0a8428f9f1084740e627a88]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:13.315217 3133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c4c06fe0ea746239f6462fbbdd45a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:13.338739 3057 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9c9757227d40b8913e577616004b6e (DEFAULT_TABLE table=ksck-test-table [id=59c4365bf0a8428f9f1084740e627a88]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:13.340806 3057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9c9757227d40b8913e577616004b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:13.342924 3056 tablet_service.cc:1467] Processing CreateTablet for tablet fcff98b6f57140d38ebb0a5b12186797 (DEFAULT_TABLE table=ksck-test-table [id=59c4365bf0a8428f9f1084740e627a88]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:13.344230 3056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcff98b6f57140d38ebb0a5b12186797. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:13.358484 3055 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4c06fe0ea746239f6462fbbdd45a4e (DEFAULT_TABLE table=ksck-test-table [id=59c4365bf0a8428f9f1084740e627a88]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:13.359840 3055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c4c06fe0ea746239f6462fbbdd45a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:13.388157 3282 tablet_bootstrap.cc:492] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a: Bootstrap starting.
I20250115 02:52:13.393904 3282 tablet_bootstrap.cc:654] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:13.396422 3283 tablet_bootstrap.cc:492] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b: Bootstrap starting.
I20250115 02:52:13.403651 3283 tablet_bootstrap.cc:654] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:13.407202 3281 tablet_bootstrap.cc:492] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8: Bootstrap starting.
I20250115 02:52:13.413231 3282 tablet_bootstrap.cc:492] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a: No bootstrap required, opened a new log
I20250115 02:52:13.413746 3282 ts_tablet_manager.cc:1397] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.010s
I20250115 02:52:13.416805 3283 tablet_bootstrap.cc:492] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b: No bootstrap required, opened a new log
I20250115 02:52:13.417330 3283 ts_tablet_manager.cc:1397] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.010s
I20250115 02:52:13.420840 3283 raft_consensus.cc:357] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.421619 3283 raft_consensus.cc:383] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:13.421943 3283 raft_consensus.cc:738] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f47f62dbab4a96a62923bb11eec90b, State: Initialized, Role: FOLLOWER
I20250115 02:52:13.422737 3283 consensus_queue.cc:260] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.427556 3282 raft_consensus.cc:357] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.428439 3282 raft_consensus.cc:383] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:13.428889 3281 tablet_bootstrap.cc:654] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:13.432691 3283 ts_tablet_manager.cc:1428] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250115 02:52:13.433874 3283 tablet_bootstrap.cc:492] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b: Bootstrap starting.
I20250115 02:52:13.428807 3282 raft_consensus.cc:738] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c80c0bbf6924856bac7d2e5cb49434a, State: Initialized, Role: FOLLOWER
I20250115 02:52:13.447088 3282 consensus_queue.cc:260] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.454207 3283 tablet_bootstrap.cc:654] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:13.459736 3281 tablet_bootstrap.cc:492] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8: No bootstrap required, opened a new log
I20250115 02:52:13.460271 3281 ts_tablet_manager.cc:1397] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.022s
I20250115 02:52:13.462924 3281 raft_consensus.cc:357] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } }
I20250115 02:52:13.463661 3281 raft_consensus.cc:383] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:13.463955 3281 raft_consensus.cc:738] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdee8bc1ad94b7f9db7c526a7615bd8, State: Initialized, Role: FOLLOWER
I20250115 02:52:13.464601 3281 consensus_queue.cc:260] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } }
I20250115 02:52:13.467621 3283 tablet_bootstrap.cc:492] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b: No bootstrap required, opened a new log
I20250115 02:52:13.472090 3283 ts_tablet_manager.cc:1397] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.000s
I20250115 02:52:13.469918 3282 ts_tablet_manager.cc:1428] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a: Time spent starting tablet: real 0.055s user 0.005s sys 0.007s
I20250115 02:52:13.474740 3282 tablet_bootstrap.cc:492] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a: Bootstrap starting.
I20250115 02:52:13.476704 3283 raft_consensus.cc:357] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.477607 3283 raft_consensus.cc:383] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:13.477973 3283 raft_consensus.cc:738] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f47f62dbab4a96a62923bb11eec90b, State: Initialized, Role: FOLLOWER
I20250115 02:52:13.480589 3283 consensus_queue.cc:260] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.482967 3283 ts_tablet_manager.cc:1428] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250115 02:52:13.484056 3283 tablet_bootstrap.cc:492] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b: Bootstrap starting.
I20250115 02:52:13.480460 3282 tablet_bootstrap.cc:654] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:13.491859 3291 raft_consensus.cc:491] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:13.494037 3291 raft_consensus.cc:513] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } }
I20250115 02:52:13.493572 3281 ts_tablet_manager.cc:1428] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8: Time spent starting tablet: real 0.033s user 0.007s sys 0.004s
I20250115 02:52:13.496623 3283 tablet_bootstrap.cc:654] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:13.498265 3281 tablet_bootstrap.cc:492] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8: Bootstrap starting.
I20250115 02:52:13.504290 3281 tablet_bootstrap.cc:654] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:13.506788 3291 leader_election.cc:290] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c80c0bbf6924856bac7d2e5cb49434a (127.2.36.67:41755), e0f47f62dbab4a96a62923bb11eec90b (127.2.36.65:37049)
I20250115 02:52:13.510645 3282 tablet_bootstrap.cc:492] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a: No bootstrap required, opened a new log
I20250115 02:52:13.511322 3282 ts_tablet_manager.cc:1397] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.000s
I20250115 02:52:13.514025 3282 raft_consensus.cc:357] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.515013 3282 raft_consensus.cc:383] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:13.515357 3282 raft_consensus.cc:738] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c80c0bbf6924856bac7d2e5cb49434a, State: Initialized, Role: FOLLOWER
I20250115 02:52:13.515575 3283 tablet_bootstrap.cc:492] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b: No bootstrap required, opened a new log
I20250115 02:52:13.516072 3283 ts_tablet_manager.cc:1397] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.006s
I20250115 02:52:13.517537 3282 consensus_queue.cc:260] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.522832 3282 ts_tablet_manager.cc:1428] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250115 02:52:13.522656 3283 raft_consensus.cc:357] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } }
I20250115 02:52:13.523530 3283 raft_consensus.cc:383] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:13.523917 3282 tablet_bootstrap.cc:492] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a: Bootstrap starting.
I20250115 02:52:13.523890 3283 raft_consensus.cc:738] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f47f62dbab4a96a62923bb11eec90b, State: Initialized, Role: FOLLOWER
I20250115 02:52:13.524863 3283 consensus_queue.cc:260] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } }
I20250115 02:52:13.527370 3283 ts_tablet_manager.cc:1428] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20250115 02:52:13.533967 3282 tablet_bootstrap.cc:654] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:13.536931 3003 catalog_manager.cc:1260] Loaded cluster ID: 995713105d22453daa382da688b1116c
I20250115 02:52:13.537576 3281 tablet_bootstrap.cc:492] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8: No bootstrap required, opened a new log
I20250115 02:52:13.545892 3281 ts_tablet_manager.cc:1397] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.004s
I20250115 02:52:13.548527 3281 raft_consensus.cc:357] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.549234 3281 raft_consensus.cc:383] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:13.549430 3020 catalog_manager.cc:1260] Loaded cluster ID: 995713105d22453daa382da688b1116c
I20250115 02:52:13.549508 3281 raft_consensus.cc:738] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdee8bc1ad94b7f9db7c526a7615bd8, State: Initialized, Role: FOLLOWER
I20250115 02:52:13.549809 3020 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473: loading cluster ID for follower catalog manager: success
I20250115 02:52:13.550305 3281 consensus_queue.cc:260] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.554723 3003 catalog_manager.cc:1553] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a: loading cluster ID for follower catalog manager: success
I20250115 02:52:13.557137 3020 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473: acquiring CA information for follower catalog manager: success
I20250115 02:52:13.552871 3281 ts_tablet_manager.cc:1428] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250115 02:52:13.564033 3281 tablet_bootstrap.cc:492] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8: Bootstrap starting.
I20250115 02:52:13.567094 3003 catalog_manager.cc:1575] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a: acquiring CA information for follower catalog manager: success
I20250115 02:52:13.569891 3281 tablet_bootstrap.cc:654] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:13.562930 3020 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:13.587039 3003 catalog_manager.cc:1603] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:13.590097 3282 tablet_bootstrap.cc:492] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a: No bootstrap required, opened a new log
I20250115 02:52:13.590655 3282 ts_tablet_manager.cc:1397] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a: Time spent bootstrapping tablet: real 0.067s user 0.011s sys 0.013s
I20250115 02:52:13.593483 3282 raft_consensus.cc:357] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } }
I20250115 02:52:13.594238 3282 raft_consensus.cc:383] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:13.594583 3282 raft_consensus.cc:738] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c80c0bbf6924856bac7d2e5cb49434a, State: Initialized, Role: FOLLOWER
I20250115 02:52:13.595315 3282 consensus_queue.cc:260] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } }
I20250115 02:52:13.600320 3282 ts_tablet_manager.cc:1428] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a: Time spent starting tablet: real 0.009s user 0.004s sys 0.002s
I20250115 02:52:13.616168 3290 raft_consensus.cc:491] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:13.616102 3227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b9c9757227d40b8913e577616004b6e" candidate_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" is_pre_election: true
I20250115 02:52:13.616930 3227 raft_consensus.cc:2463] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcdee8bc1ad94b7f9db7c526a7615bd8 in term 0.
I20250115 02:52:13.616777 3290 raft_consensus.cc:513] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.619298 3290 leader_election.cc:290] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0f47f62dbab4a96a62923bb11eec90b (127.2.36.65:37049), 1c80c0bbf6924856bac7d2e5cb49434a (127.2.36.67:41755)
I20250115 02:52:13.621739 3227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcff98b6f57140d38ebb0a5b12186797" candidate_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" is_pre_election: true
I20250115 02:52:13.622511 3227 raft_consensus.cc:2463] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcdee8bc1ad94b7f9db7c526a7615bd8 in term 0.
I20250115 02:52:13.621244 3110 leader_election.cc:304] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c80c0bbf6924856bac7d2e5cb49434a, dcdee8bc1ad94b7f9db7c526a7615bd8; no voters:
I20250115 02:52:13.627744 3290 raft_consensus.cc:2798] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:13.628264 3290 raft_consensus.cc:491] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:13.628700 3290 raft_consensus.cc:3054] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:13.629724 3110 leader_election.cc:304] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c80c0bbf6924856bac7d2e5cb49434a, dcdee8bc1ad94b7f9db7c526a7615bd8; no voters:
I20250115 02:52:13.633463 3291 raft_consensus.cc:2798] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:13.634253 3291 raft_consensus.cc:491] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:13.634613 3291 raft_consensus.cc:3054] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:13.640914 3067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b9c9757227d40b8913e577616004b6e" candidate_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f47f62dbab4a96a62923bb11eec90b" is_pre_election: true
I20250115 02:52:13.641091 3066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcff98b6f57140d38ebb0a5b12186797" candidate_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f47f62dbab4a96a62923bb11eec90b" is_pre_election: true
I20250115 02:52:13.641799 3067 raft_consensus.cc:2463] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcdee8bc1ad94b7f9db7c526a7615bd8 in term 0.
I20250115 02:52:13.641819 3066 raft_consensus.cc:2463] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcdee8bc1ad94b7f9db7c526a7615bd8 in term 0.
I20250115 02:52:13.646109 3291 raft_consensus.cc:513] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.654999 3067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcff98b6f57140d38ebb0a5b12186797" candidate_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f47f62dbab4a96a62923bb11eec90b"
I20250115 02:52:13.655635 3067 raft_consensus.cc:3054] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:13.659339 3290 raft_consensus.cc:513] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } }
I20250115 02:52:13.664386 3290 leader_election.cc:290] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [CANDIDATE]: Term 1 election: Requested vote from peers 1c80c0bbf6924856bac7d2e5cb49434a (127.2.36.67:41755), e0f47f62dbab4a96a62923bb11eec90b (127.2.36.65:37049)
I20250115 02:52:13.664775 3066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b9c9757227d40b8913e577616004b6e" candidate_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f47f62dbab4a96a62923bb11eec90b"
I20250115 02:52:13.665426 3066 raft_consensus.cc:3054] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:13.668366 3067 raft_consensus.cc:2463] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcdee8bc1ad94b7f9db7c526a7615bd8 in term 1.
I20250115 02:52:13.669575 3108 leader_election.cc:304] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcdee8bc1ad94b7f9db7c526a7615bd8, e0f47f62dbab4a96a62923bb11eec90b; no voters:
I20250115 02:52:13.671458 3227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b9c9757227d40b8913e577616004b6e" candidate_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c80c0bbf6924856bac7d2e5cb49434a"
I20250115 02:52:13.672062 3227 raft_consensus.cc:3054] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:13.673556 3290 raft_consensus.cc:2798] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:13.674043 3281 tablet_bootstrap.cc:492] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8: No bootstrap required, opened a new log
I20250115 02:52:13.674533 3281 ts_tablet_manager.cc:1397] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8: Time spent bootstrapping tablet: real 0.111s user 0.050s sys 0.034s
I20250115 02:52:13.671866 3291 leader_election.cc:290] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [CANDIDATE]: Term 1 election: Requested vote from peers e0f47f62dbab4a96a62923bb11eec90b (127.2.36.65:37049), 1c80c0bbf6924856bac7d2e5cb49434a (127.2.36.67:41755)
I20250115 02:52:13.677448 3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcff98b6f57140d38ebb0a5b12186797" candidate_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c80c0bbf6924856bac7d2e5cb49434a"
I20250115 02:52:13.678097 3226 raft_consensus.cc:3054] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:13.680099 3281 raft_consensus.cc:357] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.680837 3281 raft_consensus.cc:383] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:13.681133 3281 raft_consensus.cc:738] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdee8bc1ad94b7f9db7c526a7615bd8, State: Initialized, Role: FOLLOWER
I20250115 02:52:13.681836 3281 consensus_queue.cc:260] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.685072 3281 ts_tablet_manager.cc:1428] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
I20250115 02:52:13.685940 3066 raft_consensus.cc:2463] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcdee8bc1ad94b7f9db7c526a7615bd8 in term 1.
I20250115 02:52:13.686913 3108 leader_election.cc:304] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcdee8bc1ad94b7f9db7c526a7615bd8, e0f47f62dbab4a96a62923bb11eec90b; no voters:
I20250115 02:52:13.687773 3291 raft_consensus.cc:2798] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:13.688259 3291 raft_consensus.cc:695] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 LEADER]: Becoming Leader. State: Replica: dcdee8bc1ad94b7f9db7c526a7615bd8, State: Running, Role: LEADER
I20250115 02:52:13.689092 3291 consensus_queue.cc:237] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } }
I20250115 02:52:13.699028 3227 raft_consensus.cc:2463] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcdee8bc1ad94b7f9db7c526a7615bd8 in term 1.
I20250115 02:52:13.691291 3226 raft_consensus.cc:2463] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcdee8bc1ad94b7f9db7c526a7615bd8 in term 1.
I20250115 02:52:13.706100 3290 raft_consensus.cc:695] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 LEADER]: Becoming Leader. State: Replica: dcdee8bc1ad94b7f9db7c526a7615bd8, State: Running, Role: LEADER
I20250115 02:52:13.706983 3290 consensus_queue.cc:237] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.719381 2874 catalog_manager.cc:5526] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcdee8bc1ad94b7f9db7c526a7615bd8 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:13.744504 2874 catalog_manager.cc:5526] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcdee8bc1ad94b7f9db7c526a7615bd8 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:13.889382 3290 raft_consensus.cc:491] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:13.889796 3290 raft_consensus.cc:513] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.891566 3290 leader_election.cc:290] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0f47f62dbab4a96a62923bb11eec90b (127.2.36.65:37049), 1c80c0bbf6924856bac7d2e5cb49434a (127.2.36.67:41755)
I20250115 02:52:13.892046 3066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4c06fe0ea746239f6462fbbdd45a4e" candidate_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f47f62dbab4a96a62923bb11eec90b" is_pre_election: true
I20250115 02:52:13.892622 3066 raft_consensus.cc:2463] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcdee8bc1ad94b7f9db7c526a7615bd8 in term 0.
I20250115 02:52:13.892699 3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4c06fe0ea746239f6462fbbdd45a4e" candidate_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" is_pre_election: true
I20250115 02:52:13.893368 3226 raft_consensus.cc:2463] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcdee8bc1ad94b7f9db7c526a7615bd8 in term 0.
I20250115 02:52:13.893661 3108 leader_election.cc:304] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcdee8bc1ad94b7f9db7c526a7615bd8, e0f47f62dbab4a96a62923bb11eec90b; no voters:
I20250115 02:52:13.894582 3290 raft_consensus.cc:2798] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:13.894961 3290 raft_consensus.cc:491] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:13.895278 3290 raft_consensus.cc:3054] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:13.905320 3290 raft_consensus.cc:513] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.907830 3066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4c06fe0ea746239f6462fbbdd45a4e" candidate_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f47f62dbab4a96a62923bb11eec90b"
I20250115 02:52:13.908319 3290 leader_election.cc:290] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [CANDIDATE]: Term 1 election: Requested vote from peers e0f47f62dbab4a96a62923bb11eec90b (127.2.36.65:37049), 1c80c0bbf6924856bac7d2e5cb49434a (127.2.36.67:41755)
I20250115 02:52:13.908463 3066 raft_consensus.cc:3054] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:13.909715 3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4c06fe0ea746239f6462fbbdd45a4e" candidate_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c80c0bbf6924856bac7d2e5cb49434a"
I20250115 02:52:13.910321 3226 raft_consensus.cc:3054] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:13.917901 3066 raft_consensus.cc:2463] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcdee8bc1ad94b7f9db7c526a7615bd8 in term 1.
I20250115 02:52:13.919235 3108 leader_election.cc:304] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcdee8bc1ad94b7f9db7c526a7615bd8, e0f47f62dbab4a96a62923bb11eec90b; no voters:
I20250115 02:52:13.919968 3290 raft_consensus.cc:2798] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:13.920377 3290 raft_consensus.cc:695] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 LEADER]: Becoming Leader. State: Replica: dcdee8bc1ad94b7f9db7c526a7615bd8, State: Running, Role: LEADER
I20250115 02:52:13.921211 3290 consensus_queue.cc:237] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } }
I20250115 02:52:13.922061 3226 raft_consensus.cc:2463] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcdee8bc1ad94b7f9db7c526a7615bd8 in term 1.
I20250115 02:52:13.936829 2874 catalog_manager.cc:5526] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcdee8bc1ad94b7f9db7c526a7615bd8 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcdee8bc1ad94b7f9db7c526a7615bd8" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:14.098518 3290 consensus_queue.cc:1035] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:14.101907 3290 consensus_queue.cc:1035] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:14.128087 3290 consensus_queue.cc:1035] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:14.146831 3291 consensus_queue.cc:1035] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:14.310889 3290 consensus_queue.cc:1035] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c80c0bbf6924856bac7d2e5cb49434a" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:14.343330 3291 consensus_queue.cc:1035] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0f47f62dbab4a96a62923bb11eec90b" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:14.773367 3216 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:52:14.794862 3055 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:52:14.819144 3135 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250115 02:52:15.153610 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
W20250115 02:52:15.204116 3108 consensus_peers.cc:487] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 -> Peer e0f47f62dbab4a96a62923bb11eec90b (127.2.36.65:37049): Couldn't send request to peer e0f47f62dbab4a96a62923bb11eec90b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:15.205951 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:15.206504 2193 tablet_replica.cc:331] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b: stopping tablet replica
I20250115 02:52:15.207113 2193 raft_consensus.cc:2238] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:15.207566 2193 raft_consensus.cc:2267] T 7c4c06fe0ea746239f6462fbbdd45a4e P e0f47f62dbab4a96a62923bb11eec90b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:15.209668 2193 tablet_replica.cc:331] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b: stopping tablet replica
I20250115 02:52:15.210289 2193 raft_consensus.cc:2238] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:15.210858 2193 raft_consensus.cc:2267] T 2b9c9757227d40b8913e577616004b6e P e0f47f62dbab4a96a62923bb11eec90b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:15.212596 2193 tablet_replica.cc:331] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b: stopping tablet replica
I20250115 02:52:15.213018 2193 raft_consensus.cc:2238] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:15.213521 2193 raft_consensus.cc:2267] T fcff98b6f57140d38ebb0a5b12186797 P e0f47f62dbab4a96a62923bb11eec90b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:15.236649 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:52:15.250093 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:52:15.280205 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:15.281034 2193 tablet_replica.cc:331] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8: stopping tablet replica
I20250115 02:52:15.281642 2193 raft_consensus.cc:2238] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:15.282364 2193 raft_consensus.cc:2267] T 7c4c06fe0ea746239f6462fbbdd45a4e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 02:52:15.282804 3108 consensus_peers.cc:487] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 -> Peer e0f47f62dbab4a96a62923bb11eec90b (127.2.36.65:37049): Couldn't send request to peer e0f47f62dbab4a96a62923bb11eec90b. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:37049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:15.284660 2193 tablet_replica.cc:331] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8: stopping tablet replica
I20250115 02:52:15.285261 2193 raft_consensus.cc:2238] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:15.285904 2193 raft_consensus.cc:2267] T fcff98b6f57140d38ebb0a5b12186797 P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:15.287464 2193 tablet_replica.cc:331] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8: stopping tablet replica
I20250115 02:52:15.287892 2193 raft_consensus.cc:2238] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:15.288522 2193 raft_consensus.cc:2267] T 2b9c9757227d40b8913e577616004b6e P dcdee8bc1ad94b7f9db7c526a7615bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:15.310688 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:52:15.320526 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:52:15.346589 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:15.347147 2193 tablet_replica.cc:331] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a: stopping tablet replica
I20250115 02:52:15.347664 2193 raft_consensus.cc:2238] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:15.348039 2193 raft_consensus.cc:2267] T 7c4c06fe0ea746239f6462fbbdd45a4e P 1c80c0bbf6924856bac7d2e5cb49434a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:15.350991 2193 tablet_replica.cc:331] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a: stopping tablet replica
I20250115 02:52:15.351447 2193 raft_consensus.cc:2238] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:15.351768 2193 raft_consensus.cc:2267] T 2b9c9757227d40b8913e577616004b6e P 1c80c0bbf6924856bac7d2e5cb49434a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:15.353232 2193 tablet_replica.cc:331] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a: stopping tablet replica
I20250115 02:52:15.353644 2193 raft_consensus.cc:2238] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:15.353961 2193 raft_consensus.cc:2267] T fcff98b6f57140d38ebb0a5b12186797 P 1c80c0bbf6924856bac7d2e5cb49434a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:15.372939 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:52:15.382215 2193 master.cc:537] Master@127.2.36.126:44129 shutting down...
I20250115 02:52:15.399581 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:15.400030 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:15.400274 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P f79449208ad84e708353086b800a3b9a: stopping tablet replica
I20250115 02:52:15.418315 2193 master.cc:559] Master@127.2.36.126:44129 shutdown complete.
I20250115 02:52:15.428263 2193 master.cc:537] Master@127.2.36.125:40933 shutting down...
I20250115 02:52:15.442696 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:15.443504 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:15.443796 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6cf4710fe9f9489eb610430ea9fc77d9: stopping tablet replica
I20250115 02:52:15.462357 2193 master.cc:559] Master@127.2.36.125:40933 shutdown complete.
I20250115 02:52:15.472512 2193 master.cc:537] Master@127.2.36.124:36933 shutting down...
I20250115 02:52:15.486523 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:15.487027 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:15.487283 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 98d67f98112b4b1d9d1a849c730c3473: stopping tablet replica
I20250115 02:52:15.504626 2193 master.cc:559] Master@127.2.36.124:36933 shutdown complete.
[ OK ] RemoteKsckTest.TestCheckUnusualFlags (3331 ms)
[ RUN ] RemoteKsckTest.TestTabletServerMismatchedUUID
I20250115 02:52:15.531983 2193 test_util.cc:274] Using random seed: -1008774212
I20250115 02:52:15.532873 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:43075,127.2.36.125:44051,127.2.36.124:46361
I20250115 02:52:15.534016 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:15.539350 3353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:15.540280 3354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:15.540951 3356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:15.541707 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:15.542387 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:15.542554 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:15.542688 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909535542673 us; error 0 us; skew 500 ppm
I20250115 02:52:15.543358 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:15.545521 2193 webserver.cc:458] Webserver started at http://127.2.36.126:46299/ using document root <none> and password file <none>
I20250115 02:52:15.545955 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:15.546111 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:15.546353 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:15.547396 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "18efeacf16b84ef180b17a4496a6a46b"
format_stamp: "Formatted at 2025-01-15 02:52:15 on dist-test-slave-1lv2"
I20250115 02:52:15.551426 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20250115 02:52:15.554319 3361 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:15.555022 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250115 02:52:15.555284 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "18efeacf16b84ef180b17a4496a6a46b"
format_stamp: "Formatted at 2025-01-15 02:52:15 on dist-test-slave-1lv2"
I20250115 02:52:15.555541 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:15.576491 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:15.577483 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:15.609617 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:43075
I20250115 02:52:15.609680 3412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:43075 every 8 connection(s)
I20250115 02:52:15.613101 3413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:15.613267 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:15.619117 3415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:15.619273 3413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:15.620098 3416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:15.621727 3418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:15.622469 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:15.623281 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:15.623483 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:15.623638 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909535623622 us; error 0 us; skew 500 ppm
I20250115 02:52:15.624109 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:15.626791 2193 webserver.cc:458] Webserver started at http://127.2.36.125:35353/ using document root <none> and password file <none>
I20250115 02:52:15.627313 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:15.627477 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:15.627714 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:15.628947 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "a3d2c87272ae4171982243b0742b130d"
format_stamp: "Formatted at 2025-01-15 02:52:15 on dist-test-slave-1lv2"
I20250115 02:52:15.631685 3413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:15.634042 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
W20250115 02:52:15.636497 3413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:44051: Network error: Client connection negotiation failed: client connection to 127.2.36.125:44051: connect: Connection refused (error 111)
I20250115 02:52:15.637859 3426 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:15.638561 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:52:15.638864 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "a3d2c87272ae4171982243b0742b130d"
format_stamp: "Formatted at 2025-01-15 02:52:15 on dist-test-slave-1lv2"
I20250115 02:52:15.639132 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:15.664754 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:15.665727 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:15.682991 3413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } attempt: 1
W20250115 02:52:15.687382 3413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:44051: Network error: Client connection negotiation failed: client connection to 127.2.36.125:44051: connect: Connection refused (error 111)
I20250115 02:52:15.700098 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:44051
I20250115 02:52:15.700177 3477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:44051 every 8 connection(s)
I20250115 02:52:15.703619 3478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:15.703755 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:15.709817 3480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:15.709813 3478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:15.710398 3481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:15.711809 3483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:15.712704 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:15.713606 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:15.713856 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:15.714048 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909535714031 us; error 0 us; skew 500 ppm
I20250115 02:52:15.714619 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:15.717190 2193 webserver.cc:458] Webserver started at http://127.2.36.124:34781/ using document root <none> and password file <none>
I20250115 02:52:15.717633 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:15.717809 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:15.718044 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:15.719092 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "468df4ed46084aeea2cee0acdbad70a1"
format_stamp: "Formatted at 2025-01-15 02:52:15 on dist-test-slave-1lv2"
I20250115 02:52:15.720994 3478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:15.724228 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250115 02:52:15.728360 3490 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:15.729225 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250115 02:52:15.729502 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "468df4ed46084aeea2cee0acdbad70a1"
format_stamp: "Formatted at 2025-01-15 02:52:15 on dist-test-slave-1lv2"
I20250115 02:52:15.729773 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:15.731979 3478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:15.737295 3478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:46361: Network error: Client connection negotiation failed: client connection to 127.2.36.124:46361: connect: Connection refused (error 111)
I20250115 02:52:15.756793 3413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } attempt: 2
I20250115 02:52:15.757719 3478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } attempt: 1
W20250115 02:52:15.761967 3478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:46361: Network error: Client connection negotiation failed: client connection to 127.2.36.124:46361: connect: Connection refused (error 111)
I20250115 02:52:15.765478 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:15.766410 3413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:15.766674 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250115 02:52:15.770534 3413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:46361: Network error: Client connection negotiation failed: client connection to 127.2.36.124:46361: connect: Connection refused (error 111)
I20250115 02:52:15.794971 3413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } attempt: 1
W20250115 02:52:15.799516 3413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:46361: Network error: Client connection negotiation failed: client connection to 127.2.36.124:46361: connect: Connection refused (error 111)
I20250115 02:52:15.801560 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:46361
I20250115 02:52:15.801647 3542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:46361 every 8 connection(s)
I20250115 02:52:15.804416 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:52:15.804456 3478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } attempt: 2
I20250115 02:52:15.805330 3543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:15.814422 3543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:15.821568 3478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d: Bootstrap starting.
I20250115 02:52:15.825023 3543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:15.826864 3478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:15.831084 3478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d: No bootstrap required, opened a new log
I20250115 02:52:15.833539 3478 raft_consensus.cc:357] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } }
I20250115 02:52:15.834303 3478 raft_consensus.cc:383] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:15.834604 3478 raft_consensus.cc:738] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d2c87272ae4171982243b0742b130d, State: Initialized, Role: FOLLOWER
I20250115 02:52:15.834661 3543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:15.835393 3478 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } }
I20250115 02:52:15.837356 3548 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } } }
I20250115 02:52:15.838136 3548 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:15.838974 3478 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:15.850018 3413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } attempt: 2
I20250115 02:52:15.851850 3543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1: Bootstrap starting.
W20250115 02:52:15.854501 3559 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:15.854812 3559 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:15.857152 3543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:15.861794 3543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1: No bootstrap required, opened a new log
I20250115 02:52:15.863694 3543 raft_consensus.cc:357] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } }
I20250115 02:52:15.864208 3543 raft_consensus.cc:383] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:15.864432 3543 raft_consensus.cc:738] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468df4ed46084aeea2cee0acdbad70a1, State: Initialized, Role: FOLLOWER
I20250115 02:52:15.865155 3413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b: Bootstrap starting.
I20250115 02:52:15.864920 3543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } }
I20250115 02:52:15.866732 3561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } } }
I20250115 02:52:15.867547 3561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:15.867851 3543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:15.869964 3413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:15.874681 3413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b: No bootstrap required, opened a new log
I20250115 02:52:15.876938 3413 raft_consensus.cc:357] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } }
I20250115 02:52:15.877781 3413 raft_consensus.cc:383] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:15.878108 3413 raft_consensus.cc:738] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18efeacf16b84ef180b17a4496a6a46b, State: Initialized, Role: FOLLOWER
I20250115 02:52:15.878788 3413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } }
I20250115 02:52:15.880587 3574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } } }
I20250115 02:52:15.881156 3574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:15.882305 3413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [sys.catalog]: configured and running, proceeding with master startup.
W20250115 02:52:15.883723 3573 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:15.884058 3573 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:15.892195 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:52:15.892444 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:52:15.893683 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:15.895119 3585 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:15.895395 3585 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:52:15.900346 3586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:15.901356 3587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:15.901985 3589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:15.906687 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:15.907843 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:15.908054 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:15.908185 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909535908174 us; error 0 us; skew 500 ppm
I20250115 02:52:15.908634 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:15.910876 2193 webserver.cc:458] Webserver started at http://127.2.36.65:40907/ using document root <none> and password file <none>
I20250115 02:52:15.911309 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:15.911458 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:15.911665 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:15.912609 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "027c4024e256416fb957e0045cea277e"
format_stamp: "Formatted at 2025-01-15 02:52:15 on dist-test-slave-1lv2"
I20250115 02:52:15.916673 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:52:15.919749 3594 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:15.920460 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:52:15.920742 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "027c4024e256416fb957e0045cea277e"
format_stamp: "Formatted at 2025-01-15 02:52:15 on dist-test-slave-1lv2"
I20250115 02:52:15.921005 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:15.929559 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:15.930493 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:15.931911 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:15.933435 3561 raft_consensus.cc:491] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:15.934177 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:15.933956 3561 raft_consensus.cc:513] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } }
I20250115 02:52:15.934459 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:15.934813 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:15.935029 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:15.936081 3561 leader_election.cc:290] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18efeacf16b84ef180b17a4496a6a46b (127.2.36.126:43075), a3d2c87272ae4171982243b0742b130d (127.2.36.125:44051)
I20250115 02:52:15.936848 3388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "468df4ed46084aeea2cee0acdbad70a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18efeacf16b84ef180b17a4496a6a46b" is_pre_election: true
I20250115 02:52:15.937129 3453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "468df4ed46084aeea2cee0acdbad70a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d2c87272ae4171982243b0742b130d" is_pre_election: true
I20250115 02:52:15.937428 3388 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 468df4ed46084aeea2cee0acdbad70a1 in term 0.
I20250115 02:52:15.937834 3453 raft_consensus.cc:2463] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 468df4ed46084aeea2cee0acdbad70a1 in term 0.
I20250115 02:52:15.938329 3493 leader_election.cc:304] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18efeacf16b84ef180b17a4496a6a46b, 468df4ed46084aeea2cee0acdbad70a1; no voters:
I20250115 02:52:15.939028 3561 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:15.939296 3561 raft_consensus.cc:491] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:15.939553 3561 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:15.944499 3561 raft_consensus.cc:513] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } }
I20250115 02:52:15.946547 3561 leader_election.cc:290] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [CANDIDATE]: Term 1 election: Requested vote from peers 18efeacf16b84ef180b17a4496a6a46b (127.2.36.126:43075), a3d2c87272ae4171982243b0742b130d (127.2.36.125:44051)
I20250115 02:52:15.947129 3388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "468df4ed46084aeea2cee0acdbad70a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18efeacf16b84ef180b17a4496a6a46b"
I20250115 02:52:15.947142 3453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "468df4ed46084aeea2cee0acdbad70a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d2c87272ae4171982243b0742b130d"
I20250115 02:52:15.947703 3388 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:15.947922 3453 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:15.952075 3388 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 468df4ed46084aeea2cee0acdbad70a1 in term 1.
I20250115 02:52:15.953119 3493 leader_election.cc:304] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18efeacf16b84ef180b17a4496a6a46b, 468df4ed46084aeea2cee0acdbad70a1; no voters:
I20250115 02:52:15.953917 3561 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:15.954804 3453 raft_consensus.cc:2463] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 468df4ed46084aeea2cee0acdbad70a1 in term 1.
I20250115 02:52:15.955349 3561 raft_consensus.cc:695] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 1 LEADER]: Becoming Leader. State: Replica: 468df4ed46084aeea2cee0acdbad70a1, State: Running, Role: LEADER
I20250115 02:52:15.956195 3561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } }
I20250115 02:52:15.960172 3610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 468df4ed46084aeea2cee0acdbad70a1. Latest consensus state: current_term: 1 leader_uuid: "468df4ed46084aeea2cee0acdbad70a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } } }
I20250115 02:52:15.960945 3610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:15.962527 3619 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:52:15.967228 3619 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:52:15.976050 3453 raft_consensus.cc:1270] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [term 1 FOLLOWER]: Refusing update from remote peer 468df4ed46084aeea2cee0acdbad70a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:15.976689 3388 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [term 1 FOLLOWER]: Refusing update from remote peer 468df4ed46084aeea2cee0acdbad70a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:15.977676 3610 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:15.978474 3561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:15.989651 3548 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 468df4ed46084aeea2cee0acdbad70a1. Latest consensus state: current_term: 1 leader_uuid: "468df4ed46084aeea2cee0acdbad70a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } } }
I20250115 02:52:15.990514 3548 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:15.990867 3574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 468df4ed46084aeea2cee0acdbad70a1. Latest consensus state: current_term: 1 leader_uuid: "468df4ed46084aeea2cee0acdbad70a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } } }
I20250115 02:52:15.991626 3574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:15.994377 3610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "468df4ed46084aeea2cee0acdbad70a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } } }
I20250115 02:52:15.995165 3610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:15.995689 3619 catalog_manager.cc:1348] Generated new cluster ID: 16871bd46c03401eb41fe148ebb65caa
I20250115 02:52:15.995999 3619 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:52:15.997002 3548 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "468df4ed46084aeea2cee0acdbad70a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } } }
I20250115 02:52:15.997761 3548 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:16.039016 3610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "468df4ed46084aeea2cee0acdbad70a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } } }
I20250115 02:52:16.042807 3610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:16.052902 3574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "468df4ed46084aeea2cee0acdbad70a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18efeacf16b84ef180b17a4496a6a46b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43075 } } peers { permanent_uuid: "a3d2c87272ae4171982243b0742b130d" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 44051 } } peers { permanent_uuid: "468df4ed46084aeea2cee0acdbad70a1" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46361 } } }
I20250115 02:52:16.054279 3574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:16.079311 3619 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:52:16.081571 3619 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:52:16.093708 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:43753
I20250115 02:52:16.093748 3669 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:43753 every 8 connection(s)
I20250115 02:52:16.130157 3619 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1: Generated new TSK 0
I20250115 02:52:16.131139 3619 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:52:16.140602 3671 heartbeater.cc:346] Connected to a master server at 127.2.36.125:44051
I20250115 02:52:16.141104 3671 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:16.142195 3671 heartbeater.cc:509] Master 127.2.36.125:44051 requested a full tablet report, sending...
I20250115 02:52:16.143172 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:16.143308 3670 heartbeater.cc:346] Connected to a master server at 127.2.36.124:46361
I20250115 02:52:16.143793 3670 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:16.144824 3670 heartbeater.cc:509] Master 127.2.36.124:46361 requested a full tablet report, sending...
I20250115 02:52:16.145538 3443 ts_manager.cc:194] Registered new tserver with Master: 027c4024e256416fb957e0045cea277e (127.2.36.65:43753)
I20250115 02:52:16.147619 3508 ts_manager.cc:194] Registered new tserver with Master: 027c4024e256416fb957e0045cea277e (127.2.36.65:43753)
I20250115 02:52:16.149900 3508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46968
I20250115 02:52:16.151110 3672 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43075
I20250115 02:52:16.151546 3672 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:16.152529 3672 heartbeater.cc:509] Master 127.2.36.126:43075 requested a full tablet report, sending...
I20250115 02:52:16.156162 3378 ts_manager.cc:194] Registered new tserver with Master: 027c4024e256416fb957e0045cea277e (127.2.36.65:43753)
W20250115 02:52:16.157449 3678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:16.156219 3677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:16.159076 3680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:16.163122 3670 heartbeater.cc:501] Master 127.2.36.124:46361 was elected leader, sending a full tablet report...
I20250115 02:52:16.165956 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:16.166814 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:16.167032 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:16.167181 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909536167166 us; error 0 us; skew 500 ppm
I20250115 02:52:16.167666 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:16.170855 2193 webserver.cc:458] Webserver started at http://127.2.36.66:33397/ using document root <none> and password file <none>
I20250115 02:52:16.171554 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:16.171828 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:16.172230 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:16.173660 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "be1b14791ef64f6facefecabbae5f456"
format_stamp: "Formatted at 2025-01-15 02:52:16 on dist-test-slave-1lv2"
I20250115 02:52:16.179008 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250115 02:52:16.182466 3685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:16.183446 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:52:16.183849 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "be1b14791ef64f6facefecabbae5f456"
format_stamp: "Formatted at 2025-01-15 02:52:16 on dist-test-slave-1lv2"
I20250115 02:52:16.184132 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:16.202273 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:16.203811 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:16.205852 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:16.208230 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:16.208433 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:16.208631 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:16.208771 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:16.254911 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:34895
I20250115 02:52:16.255002 3747 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:34895 every 8 connection(s)
I20250115 02:52:16.271524 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:16.287586 3755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:16.291250 3750 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43075
I20250115 02:52:16.291746 3750 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:16.292810 3750 heartbeater.cc:509] Master 127.2.36.126:43075 requested a full tablet report, sending...
I20250115 02:52:16.293931 3749 heartbeater.cc:346] Connected to a master server at 127.2.36.125:44051
I20250115 02:52:16.294385 3749 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:16.295393 3749 heartbeater.cc:509] Master 127.2.36.125:44051 requested a full tablet report, sending...
I20250115 02:52:16.298904 3443 ts_manager.cc:194] Registered new tserver with Master: be1b14791ef64f6facefecabbae5f456 (127.2.36.66:34895)
I20250115 02:52:16.301968 3378 ts_manager.cc:194] Registered new tserver with Master: be1b14791ef64f6facefecabbae5f456 (127.2.36.66:34895)
I20250115 02:52:16.303879 3748 heartbeater.cc:346] Connected to a master server at 127.2.36.124:46361
I20250115 02:52:16.304320 3748 heartbeater.cc:463] Registering TS with master...
W20250115 02:52:16.304914 3756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:16.305545 3748 heartbeater.cc:509] Master 127.2.36.124:46361 requested a full tablet report, sending...
I20250115 02:52:16.308297 3508 ts_manager.cc:194] Registered new tserver with Master: be1b14791ef64f6facefecabbae5f456 (127.2.36.66:34895)
I20250115 02:52:16.309846 3508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46982
W20250115 02:52:16.315188 3758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:16.315507 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:16.317134 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:16.317569 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:16.317866 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909536317843 us; error 0 us; skew 500 ppm
I20250115 02:52:16.318904 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:16.321915 2193 webserver.cc:458] Webserver started at http://127.2.36.67:46283/ using document root <none> and password file <none>
I20250115 02:52:16.322551 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:16.322784 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:16.323025 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:16.324115 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "33878f6299994faf9451dc33aed46f4d"
format_stamp: "Formatted at 2025-01-15 02:52:16 on dist-test-slave-1lv2"
I20250115 02:52:16.324692 3748 heartbeater.cc:501] Master 127.2.36.124:46361 was elected leader, sending a full tablet report...
I20250115 02:52:16.330899 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250115 02:52:16.334390 3763 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:16.335206 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:52:16.335597 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "33878f6299994faf9451dc33aed46f4d"
format_stamp: "Formatted at 2025-01-15 02:52:16 on dist-test-slave-1lv2"
I20250115 02:52:16.335945 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:16.391947 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:16.394059 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:16.397312 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:16.400086 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:16.400426 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250115 02:52:16.400673 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:16.400831 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:16.463243 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:41377
I20250115 02:52:16.463510 3825 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:41377 every 8 connection(s)
I20250115 02:52:16.496008 3827 heartbeater.cc:346] Connected to a master server at 127.2.36.125:44051
I20250115 02:52:16.496513 3827 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:16.497467 3827 heartbeater.cc:509] Master 127.2.36.125:44051 requested a full tablet report, sending...
I20250115 02:52:16.500458 3443 ts_manager.cc:194] Registered new tserver with Master: 33878f6299994faf9451dc33aed46f4d (127.2.36.67:41377)
I20250115 02:52:16.508543 3829 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43075
I20250115 02:52:16.508944 3829 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:16.509472 3826 heartbeater.cc:346] Connected to a master server at 127.2.36.124:46361
I20250115 02:52:16.509855 3826 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:16.509924 3829 heartbeater.cc:509] Master 127.2.36.126:43075 requested a full tablet report, sending...
I20250115 02:52:16.510835 3826 heartbeater.cc:509] Master 127.2.36.124:46361 requested a full tablet report, sending...
I20250115 02:52:16.513126 3507 ts_manager.cc:194] Registered new tserver with Master: 33878f6299994faf9451dc33aed46f4d (127.2.36.67:41377)
I20250115 02:52:16.513893 3377 ts_manager.cc:194] Registered new tserver with Master: 33878f6299994faf9451dc33aed46f4d (127.2.36.67:41377)
I20250115 02:52:16.515002 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012694734s
I20250115 02:52:16.515058 3507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46986
I20250115 02:52:16.530567 3826 heartbeater.cc:501] Master 127.2.36.124:46361 was elected leader, sending a full tablet report...
I20250115 02:52:16.571877 3506 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46994:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:52:16.575016 3506 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:52:16.705680 3625 tablet_service.cc:1467] Processing CreateTablet for tablet bea6f3ae46a949f3a513871132a725c2 (DEFAULT_TABLE table=ksck-test-table [id=1ba9180864d5422c9ce0bd22cdc9c04e]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:16.707327 3625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bea6f3ae46a949f3a513871132a725c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:16.711105 3713 tablet_service.cc:1467] Processing CreateTablet for tablet bea6f3ae46a949f3a513871132a725c2 (DEFAULT_TABLE table=ksck-test-table [id=1ba9180864d5422c9ce0bd22cdc9c04e]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:16.711508 3712 tablet_service.cc:1467] Processing CreateTablet for tablet 59fda53cf26b4f9ca63f6fbf9dc4c91b (DEFAULT_TABLE table=ksck-test-table [id=1ba9180864d5422c9ce0bd22cdc9c04e]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:16.711752 3711 tablet_service.cc:1467] Processing CreateTablet for tablet a9363d43a0ad4f559cc84450ba686b73 (DEFAULT_TABLE table=ksck-test-table [id=1ba9180864d5422c9ce0bd22cdc9c04e]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:16.713511 3712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59fda53cf26b4f9ca63f6fbf9dc4c91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:16.717412 3624 tablet_service.cc:1467] Processing CreateTablet for tablet 59fda53cf26b4f9ca63f6fbf9dc4c91b (DEFAULT_TABLE table=ksck-test-table [id=1ba9180864d5422c9ce0bd22cdc9c04e]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:16.718081 3711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9363d43a0ad4f559cc84450ba686b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:16.718801 3624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59fda53cf26b4f9ca63f6fbf9dc4c91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:16.721972 3713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bea6f3ae46a949f3a513871132a725c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:16.732823 3623 tablet_service.cc:1467] Processing CreateTablet for tablet a9363d43a0ad4f559cc84450ba686b73 (DEFAULT_TABLE table=ksck-test-table [id=1ba9180864d5422c9ce0bd22cdc9c04e]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:16.734236 3623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9363d43a0ad4f559cc84450ba686b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:16.760547 3791 tablet_service.cc:1467] Processing CreateTablet for tablet bea6f3ae46a949f3a513871132a725c2 (DEFAULT_TABLE table=ksck-test-table [id=1ba9180864d5422c9ce0bd22cdc9c04e]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:16.762111 3791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bea6f3ae46a949f3a513871132a725c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:16.772971 3790 tablet_service.cc:1467] Processing CreateTablet for tablet 59fda53cf26b4f9ca63f6fbf9dc4c91b (DEFAULT_TABLE table=ksck-test-table [id=1ba9180864d5422c9ce0bd22cdc9c04e]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:16.775903 3789 tablet_service.cc:1467] Processing CreateTablet for tablet a9363d43a0ad4f559cc84450ba686b73 (DEFAULT_TABLE table=ksck-test-table [id=1ba9180864d5422c9ce0bd22cdc9c04e]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:16.774624 3790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59fda53cf26b4f9ca63f6fbf9dc4c91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:16.788143 3789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9363d43a0ad4f559cc84450ba686b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:16.787997 3848 tablet_bootstrap.cc:492] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e: Bootstrap starting.
I20250115 02:52:16.811811 3848 tablet_bootstrap.cc:654] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:16.838615 3847 tablet_bootstrap.cc:492] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456: Bootstrap starting.
I20250115 02:52:16.850231 3847 tablet_bootstrap.cc:654] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:16.862185 3559 catalog_manager.cc:1260] Loaded cluster ID: 16871bd46c03401eb41fe148ebb65caa
I20250115 02:52:16.865638 3559 catalog_manager.cc:1553] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d: loading cluster ID for follower catalog manager: success
I20250115 02:52:16.866847 3847 tablet_bootstrap.cc:492] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456: No bootstrap required, opened a new log
I20250115 02:52:16.867386 3847 ts_tablet_manager.cc:1397] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.009s
I20250115 02:52:16.871230 3847 raft_consensus.cc:357] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:16.872046 3847 raft_consensus.cc:383] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:16.872375 3847 raft_consensus.cc:738] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1b14791ef64f6facefecabbae5f456, State: Initialized, Role: FOLLOWER
I20250115 02:52:16.873169 3847 consensus_queue.cc:260] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:16.877177 3848 tablet_bootstrap.cc:492] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e: No bootstrap required, opened a new log
I20250115 02:52:16.877526 3559 catalog_manager.cc:1575] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d: acquiring CA information for follower catalog manager: success
I20250115 02:52:16.877713 3848 ts_tablet_manager.cc:1397] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e: Time spent bootstrapping tablet: real 0.090s user 0.012s sys 0.007s
I20250115 02:52:16.883148 3559 catalog_manager.cc:1603] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:16.882915 3848 raft_consensus.cc:357] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:16.883980 3848 raft_consensus.cc:383] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:16.884356 3848 raft_consensus.cc:738] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027c4024e256416fb957e0045cea277e, State: Initialized, Role: FOLLOWER
I20250115 02:52:16.885192 3848 consensus_queue.cc:260] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:16.888171 3847 ts_tablet_manager.cc:1428] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456: Time spent starting tablet: real 0.020s user 0.009s sys 0.001s
I20250115 02:52:16.897833 3853 tablet_bootstrap.cc:492] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d: Bootstrap starting.
I20250115 02:52:16.901225 3585 catalog_manager.cc:1260] Loaded cluster ID: 16871bd46c03401eb41fe148ebb65caa
I20250115 02:52:16.901549 3585 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b: loading cluster ID for follower catalog manager: success
I20250115 02:52:16.908552 3585 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b: acquiring CA information for follower catalog manager: success
I20250115 02:52:16.915933 3847 tablet_bootstrap.cc:492] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456: Bootstrap starting.
I20250115 02:52:16.918673 3585 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:16.922653 3853 tablet_bootstrap.cc:654] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:16.924522 3847 tablet_bootstrap.cc:654] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:16.931217 3848 ts_tablet_manager.cc:1428] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e: Time spent starting tablet: real 0.053s user 0.021s sys 0.012s
I20250115 02:52:16.932574 3848 tablet_bootstrap.cc:492] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e: Bootstrap starting.
I20250115 02:52:16.940677 3848 tablet_bootstrap.cc:654] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:16.942416 3847 tablet_bootstrap.cc:492] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456: No bootstrap required, opened a new log
I20250115 02:52:16.944670 3847 ts_tablet_manager.cc:1397] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.000s
I20250115 02:52:16.948854 3847 raft_consensus.cc:357] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:16.949751 3847 raft_consensus.cc:383] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:16.950091 3847 raft_consensus.cc:738] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1b14791ef64f6facefecabbae5f456, State: Initialized, Role: FOLLOWER
I20250115 02:52:16.950909 3847 consensus_queue.cc:260] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:16.953302 3847 ts_tablet_manager.cc:1428] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250115 02:52:16.957342 3847 tablet_bootstrap.cc:492] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456: Bootstrap starting.
I20250115 02:52:16.960085 3848 tablet_bootstrap.cc:492] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e: No bootstrap required, opened a new log
I20250115 02:52:16.960613 3848 ts_tablet_manager.cc:1397] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.003s
I20250115 02:52:16.963433 3847 tablet_bootstrap.cc:654] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:16.968189 3848 raft_consensus.cc:357] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:16.970721 3848 raft_consensus.cc:383] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:16.971048 3848 raft_consensus.cc:738] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027c4024e256416fb957e0045cea277e, State: Initialized, Role: FOLLOWER
I20250115 02:52:16.973456 3853 tablet_bootstrap.cc:492] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d: No bootstrap required, opened a new log
I20250115 02:52:16.974018 3853 ts_tablet_manager.cc:1397] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d: Time spent bootstrapping tablet: real 0.076s user 0.016s sys 0.000s
I20250115 02:52:16.980780 3853 raft_consensus.cc:357] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:16.981544 3853 raft_consensus.cc:383] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:16.981863 3853 raft_consensus.cc:738] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33878f6299994faf9451dc33aed46f4d, State: Initialized, Role: FOLLOWER
I20250115 02:52:16.982733 3848 consensus_queue.cc:260] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:16.991015 3848 ts_tablet_manager.cc:1428] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e: Time spent starting tablet: real 0.030s user 0.004s sys 0.005s
I20250115 02:52:16.991822 3853 consensus_queue.cc:260] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:16.998138 3848 tablet_bootstrap.cc:492] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e: Bootstrap starting.
I20250115 02:52:17.004904 3848 tablet_bootstrap.cc:654] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:17.013793 3847 tablet_bootstrap.cc:492] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456: No bootstrap required, opened a new log
I20250115 02:52:17.014326 3847 ts_tablet_manager.cc:1397] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.042s
I20250115 02:52:17.015064 3853 ts_tablet_manager.cc:1428] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d: Time spent starting tablet: real 0.041s user 0.006s sys 0.001s
I20250115 02:52:17.017546 3853 tablet_bootstrap.cc:492] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d: Bootstrap starting.
I20250115 02:52:17.017176 3847 raft_consensus.cc:357] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } }
I20250115 02:52:17.018018 3847 raft_consensus.cc:383] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:17.018357 3847 raft_consensus.cc:738] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1b14791ef64f6facefecabbae5f456, State: Initialized, Role: FOLLOWER
I20250115 02:52:17.019124 3847 consensus_queue.cc:260] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } }
I20250115 02:52:17.021080 3847 ts_tablet_manager.cc:1428] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20250115 02:52:17.033939 3853 tablet_bootstrap.cc:654] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:17.045060 3853 tablet_bootstrap.cc:492] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d: No bootstrap required, opened a new log
I20250115 02:52:17.045589 3853 ts_tablet_manager.cc:1397] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.004s
I20250115 02:52:17.048472 3853 raft_consensus.cc:357] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:17.049242 3853 raft_consensus.cc:383] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:17.049566 3853 raft_consensus.cc:738] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33878f6299994faf9451dc33aed46f4d, State: Initialized, Role: FOLLOWER
I20250115 02:52:17.050287 3853 consensus_queue.cc:260] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:17.055765 3848 tablet_bootstrap.cc:492] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e: No bootstrap required, opened a new log
I20250115 02:52:17.056306 3848 ts_tablet_manager.cc:1397] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.007s
I20250115 02:52:17.059222 3853 ts_tablet_manager.cc:1428] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d: Time spent starting tablet: real 0.013s user 0.001s sys 0.005s
I20250115 02:52:17.061446 3858 raft_consensus.cc:491] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:17.061965 3858 raft_consensus.cc:513] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:17.063081 3848 raft_consensus.cc:357] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } }
I20250115 02:52:17.063921 3848 raft_consensus.cc:383] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:17.064265 3848 raft_consensus.cc:738] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027c4024e256416fb957e0045cea277e, State: Initialized, Role: FOLLOWER
I20250115 02:52:17.067081 3858 leader_election.cc:290] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 027c4024e256416fb957e0045cea277e (127.2.36.65:43753), be1b14791ef64f6facefecabbae5f456 (127.2.36.66:34895)
I20250115 02:52:17.079107 3853 tablet_bootstrap.cc:492] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d: Bootstrap starting.
I20250115 02:52:17.086107 3853 tablet_bootstrap.cc:654] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:17.094331 3852 raft_consensus.cc:491] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:17.094871 3852 raft_consensus.cc:513] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:17.091218 3848 consensus_queue.cc:260] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } }
I20250115 02:52:17.097249 3852 leader_election.cc:290] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be1b14791ef64f6facefecabbae5f456 (127.2.36.66:34895), 33878f6299994faf9451dc33aed46f4d (127.2.36.67:41377)
I20250115 02:52:17.104439 3848 ts_tablet_manager.cc:1428] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e: Time spent starting tablet: real 0.048s user 0.006s sys 0.001s
I20250115 02:52:17.115342 3636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fda53cf26b4f9ca63f6fbf9dc4c91b" candidate_uuid: "33878f6299994faf9451dc33aed46f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027c4024e256416fb957e0045cea277e" is_pre_election: true
I20250115 02:52:17.116402 3636 raft_consensus.cc:2463] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33878f6299994faf9451dc33aed46f4d in term 0.
I20250115 02:52:17.117758 3765 leader_election.cc:304] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027c4024e256416fb957e0045cea277e, 33878f6299994faf9451dc33aed46f4d; no voters:
I20250115 02:52:17.118682 3858 raft_consensus.cc:2798] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:17.119074 3858 raft_consensus.cc:491] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:17.119415 3858 raft_consensus.cc:3054] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:17.124982 3723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fda53cf26b4f9ca63f6fbf9dc4c91b" candidate_uuid: "33878f6299994faf9451dc33aed46f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1b14791ef64f6facefecabbae5f456" is_pre_election: true
I20250115 02:52:17.125738 3723 raft_consensus.cc:2463] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33878f6299994faf9451dc33aed46f4d in term 0.
I20250115 02:52:17.132539 3858 raft_consensus.cc:513] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:17.135532 3636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fda53cf26b4f9ca63f6fbf9dc4c91b" candidate_uuid: "33878f6299994faf9451dc33aed46f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027c4024e256416fb957e0045cea277e"
I20250115 02:52:17.136201 3636 raft_consensus.cc:3054] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:17.151490 3636 raft_consensus.cc:2463] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33878f6299994faf9451dc33aed46f4d in term 1.
I20250115 02:52:17.152632 3723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9363d43a0ad4f559cc84450ba686b73" candidate_uuid: "027c4024e256416fb957e0045cea277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1b14791ef64f6facefecabbae5f456" is_pre_election: true
I20250115 02:52:17.152668 3765 leader_election.cc:304] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027c4024e256416fb957e0045cea277e, 33878f6299994faf9451dc33aed46f4d; no voters:
I20250115 02:52:17.153535 3723 raft_consensus.cc:2463] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 027c4024e256416fb957e0045cea277e in term 0.
I20250115 02:52:17.154871 3597 leader_election.cc:304] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027c4024e256416fb957e0045cea277e, be1b14791ef64f6facefecabbae5f456; no voters:
I20250115 02:52:17.155969 3863 raft_consensus.cc:2798] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:17.156503 3863 raft_consensus.cc:491] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:17.156886 3863 raft_consensus.cc:3054] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:17.163681 3858 leader_election.cc:290] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [CANDIDATE]: Term 1 election: Requested vote from peers 027c4024e256416fb957e0045cea277e (127.2.36.65:43753), be1b14791ef64f6facefecabbae5f456 (127.2.36.66:34895)
I20250115 02:52:17.164283 3858 raft_consensus.cc:2798] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:17.164738 3858 raft_consensus.cc:695] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 1 LEADER]: Becoming Leader. State: Replica: 33878f6299994faf9451dc33aed46f4d, State: Running, Role: LEADER
I20250115 02:52:17.165568 3858 consensus_queue.cc:237] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:17.170334 3863 raft_consensus.cc:513] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:17.173105 3723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fda53cf26b4f9ca63f6fbf9dc4c91b" candidate_uuid: "33878f6299994faf9451dc33aed46f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1b14791ef64f6facefecabbae5f456"
I20250115 02:52:17.173776 3723 raft_consensus.cc:3054] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:17.176390 3722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9363d43a0ad4f559cc84450ba686b73" candidate_uuid: "027c4024e256416fb957e0045cea277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1b14791ef64f6facefecabbae5f456"
I20250115 02:52:17.176869 3722 raft_consensus.cc:3054] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:17.179524 3801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9363d43a0ad4f559cc84450ba686b73" candidate_uuid: "027c4024e256416fb957e0045cea277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33878f6299994faf9451dc33aed46f4d" is_pre_election: true
I20250115 02:52:17.179982 3800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9363d43a0ad4f559cc84450ba686b73" candidate_uuid: "027c4024e256416fb957e0045cea277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33878f6299994faf9451dc33aed46f4d"
I20250115 02:52:17.180354 3801 raft_consensus.cc:2463] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 027c4024e256416fb957e0045cea277e in term 0.
I20250115 02:52:17.183286 3722 raft_consensus.cc:2463] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 027c4024e256416fb957e0045cea277e in term 1.
I20250115 02:52:17.185047 3863 leader_election.cc:290] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [CANDIDATE]: Term 1 election: Requested vote from peers be1b14791ef64f6facefecabbae5f456 (127.2.36.66:34895), 33878f6299994faf9451dc33aed46f4d (127.2.36.67:41377)
I20250115 02:52:17.185245 3597 leader_election.cc:304] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 027c4024e256416fb957e0045cea277e, be1b14791ef64f6facefecabbae5f456; no voters: 33878f6299994faf9451dc33aed46f4d
I20250115 02:52:17.186527 3852 raft_consensus.cc:2798] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:17.196817 3851 raft_consensus.cc:491] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:17.197360 3851 raft_consensus.cc:513] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } }
I20250115 02:52:17.200145 3851 leader_election.cc:290] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 027c4024e256416fb957e0045cea277e (127.2.36.65:43753), 33878f6299994faf9451dc33aed46f4d (127.2.36.67:41377)
I20250115 02:52:17.207281 3723 raft_consensus.cc:2463] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33878f6299994faf9451dc33aed46f4d in term 1.
I20250115 02:52:17.210194 3852 raft_consensus.cc:695] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 1 LEADER]: Becoming Leader. State: Replica: 027c4024e256416fb957e0045cea277e, State: Running, Role: LEADER
I20250115 02:52:17.214789 3852 consensus_queue.cc:237] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:17.232583 3853 tablet_bootstrap.cc:492] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d: No bootstrap required, opened a new log
I20250115 02:52:17.233045 3853 ts_tablet_manager.cc:1397] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d: Time spent bootstrapping tablet: real 0.163s user 0.021s sys 0.025s
I20250115 02:52:17.234578 3506 catalog_manager.cc:5526] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e reported cstate change: term changed from 0 to 1, leader changed from <none> to 027c4024e256416fb957e0045cea277e (127.2.36.65). New cstate: current_term: 1 leader_uuid: "027c4024e256416fb957e0045cea277e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:17.238369 3507 catalog_manager.cc:5526] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 33878f6299994faf9451dc33aed46f4d (127.2.36.67). New cstate: current_term: 1 leader_uuid: "33878f6299994faf9451dc33aed46f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } health_report { overall_health: HEALTHY } } }
I20250115 02:52:17.248380 3853 raft_consensus.cc:357] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } }
I20250115 02:52:17.249197 3853 raft_consensus.cc:383] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:17.249522 3853 raft_consensus.cc:738] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33878f6299994faf9451dc33aed46f4d, State: Initialized, Role: FOLLOWER
I20250115 02:52:17.250212 3853 consensus_queue.cc:260] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } }
I20250115 02:52:17.251786 3636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea6f3ae46a949f3a513871132a725c2" candidate_uuid: "be1b14791ef64f6facefecabbae5f456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027c4024e256416fb957e0045cea277e" is_pre_election: true
I20250115 02:52:17.252512 3636 raft_consensus.cc:2463] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1b14791ef64f6facefecabbae5f456 in term 0.
I20250115 02:52:17.253962 3687 leader_election.cc:304] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027c4024e256416fb957e0045cea277e, be1b14791ef64f6facefecabbae5f456; no voters:
I20250115 02:52:17.254861 3851 raft_consensus.cc:2798] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:17.255206 3851 raft_consensus.cc:491] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:17.255532 3851 raft_consensus.cc:3054] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:17.263260 3853 ts_tablet_manager.cc:1428] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d: Time spent starting tablet: real 0.030s user 0.005s sys 0.004s
I20250115 02:52:17.263891 3851 raft_consensus.cc:513] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } }
I20250115 02:52:17.266184 3851 leader_election.cc:290] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [CANDIDATE]: Term 1 election: Requested vote from peers 027c4024e256416fb957e0045cea277e (127.2.36.65:43753), 33878f6299994faf9451dc33aed46f4d (127.2.36.67:41377)
I20250115 02:52:17.267856 3636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea6f3ae46a949f3a513871132a725c2" candidate_uuid: "be1b14791ef64f6facefecabbae5f456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027c4024e256416fb957e0045cea277e"
I20250115 02:52:17.268533 3636 raft_consensus.cc:3054] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:17.280143 3636 raft_consensus.cc:2463] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1b14791ef64f6facefecabbae5f456 in term 1.
I20250115 02:52:17.281159 3687 leader_election.cc:304] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027c4024e256416fb957e0045cea277e, be1b14791ef64f6facefecabbae5f456; no voters:
I20250115 02:52:17.283501 3851 raft_consensus.cc:2798] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:17.294906 3851 raft_consensus.cc:695] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 1 LEADER]: Becoming Leader. State: Replica: be1b14791ef64f6facefecabbae5f456, State: Running, Role: LEADER
I20250115 02:52:17.295850 3851 consensus_queue.cc:237] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } }
I20250115 02:52:17.298982 3801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea6f3ae46a949f3a513871132a725c2" candidate_uuid: "be1b14791ef64f6facefecabbae5f456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33878f6299994faf9451dc33aed46f4d"
I20250115 02:52:17.299062 3800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea6f3ae46a949f3a513871132a725c2" candidate_uuid: "be1b14791ef64f6facefecabbae5f456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33878f6299994faf9451dc33aed46f4d" is_pre_election: true
I20250115 02:52:17.302299 3801 raft_consensus.cc:3054] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:17.315269 3801 raft_consensus.cc:2463] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1b14791ef64f6facefecabbae5f456 in term 1.
I20250115 02:52:17.357726 3506 catalog_manager.cc:5526] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 reported cstate change: term changed from 0 to 1, leader changed from <none> to be1b14791ef64f6facefecabbae5f456 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "be1b14791ef64f6facefecabbae5f456" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } health_report { overall_health: HEALTHY } } }
I20250115 02:52:17.614132 3852 consensus_queue.cc:1035] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:17.626674 3858 consensus_queue.cc:1035] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:17.631500 3801 raft_consensus.cc:3054] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:17.644873 3852 consensus_queue.cc:1035] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [LEADER]: Connected to new peer: Peer: permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:17.695142 3851 consensus_queue.cc:1035] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:17.731755 3858 consensus_queue.cc:1035] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:17.779902 3875 consensus_queue.cc:1035] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 02:52:18.200954 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
W20250115 02:52:18.220769 3765 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 11 similar messages]
W20250115 02:52:18.224848 3765 consensus_peers.cc:487] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d -> Peer 027c4024e256416fb957e0045cea277e (127.2.36.65:43753): Couldn't send request to peer 027c4024e256416fb957e0045cea277e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:18.243659 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:18.244534 2193 tablet_replica.cc:331] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e: stopping tablet replica
I20250115 02:52:18.245503 2193 raft_consensus.cc:2238] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:18.245924 2193 raft_consensus.cc:2267] T bea6f3ae46a949f3a513871132a725c2 P 027c4024e256416fb957e0045cea277e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:18.248346 2193 tablet_replica.cc:331] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e: stopping tablet replica
I20250115 02:52:18.248973 2193 raft_consensus.cc:2238] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:18.249502 2193 raft_consensus.cc:2267] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 027c4024e256416fb957e0045cea277e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:18.251889 2193 tablet_replica.cc:331] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e: stopping tablet replica
I20250115 02:52:18.252378 2193 raft_consensus.cc:2238] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:18.253023 2193 raft_consensus.cc:2267] T a9363d43a0ad4f559cc84450ba686b73 P 027c4024e256416fb957e0045cea277e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:18.281054 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:52:18.294947 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:18.302338 3931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:18.309008 3932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:18.312343 3687 consensus_peers.cc:487] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 -> Peer 027c4024e256416fb957e0045cea277e (127.2.36.65:43753): Couldn't send request to peer 027c4024e256416fb957e0045cea277e. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250115 02:52:18.315922 3935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:18.321185 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:18.323750 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:18.324040 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:18.324235 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909538324218 us; error 0 us; skew 500 ppm
I20250115 02:52:18.324750 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:18.327648 2193 webserver.cc:458] Webserver started at http://127.2.36.65:40807/ using document root <none> and password file <none>
I20250115 02:52:18.328191 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:18.328428 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:18.328732 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:18.330117 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-0-root2/instance:
uuid: "8cfd890708f24463b8f2c2e29e8eda79"
format_stamp: "Formatted at 2025-01-15 02:52:18 on dist-test-slave-1lv2"
I20250115 02:52:18.336764 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250115 02:52:18.341490 3940 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:18.342496 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250115 02:52:18.342898 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-0-root2
uuid: "8cfd890708f24463b8f2c2e29e8eda79"
format_stamp: "Formatted at 2025-01-15 02:52:18 on dist-test-slave-1lv2"
I20250115 02:52:18.343245 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-0-root2
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-0-root2
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletServerMismatchedUUID.1736909525896969-2193-0/minicluster-data/ts-0-root2/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:18.373917 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:18.375021 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:18.380342 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:18.395911 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:18.396198 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:18.396474 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:18.396687 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:18.467784 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:43753
I20250115 02:52:18.467882 4004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:43753 every 8 connection(s)
W20250115 02:52:18.499547 4005 heartbeater.cc:646] Failed to heartbeat to 127.0.0.1:7051 (0 consecutive failures): Network error: Failed to ping master at 127.0.0.1:7051: Client connection negotiation failed: client connection to 127.0.0.1:7051: connect: Connection refused (error 111)
W20250115 02:52:18.557287 4005 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250115 02:52:18.563601 3790 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:52:18.576463 3713 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20250115 02:52:18.696852 3980 tablet_service.cc:374] Invalid argument: UpdateConsensus: Wrong destination UUID requested. Local UUID: 8cfd890708f24463b8f2c2e29e8eda79. Requested UUID: 027c4024e256416fb957e0045cea277e: from {username='slave'} at 127.0.0.1:49458: tablet_id: "59fda53cf26b4f9ca63f6fbf9dc4c91b" caller_uuid: "33878f6299994faf9451dc33aed46f4d" caller_term: 1 preceding_id { term: 1 index: 1 } dest_uuid: "027c4024e256416fb957e0045cea277e" committed_index: 1 all_replicated_index: 1 safe_timestamp: 7114381470101909504 last_idx_appended_to_leader: 1
I20250115 02:52:18.702656 3765 consensus_queue.cc:579] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [LEADER]: The tablet replica hosted on peer 027c4024e256416fb957e0045cea277e has failed
I20250115 02:52:18.720880 3801 consensus_queue.cc:237] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:18.735877 3722 raft_consensus.cc:1270] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Refusing update from remote peer 33878f6299994faf9451dc33aed46f4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:18.737763 4019 consensus_queue.cc:1035] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250115 02:52:18.753240 3909 raft_consensus.cc:2949] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } }
I20250115 02:52:18.754732 3722 raft_consensus.cc:2949] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } }
I20250115 02:52:18.763583 3492 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 59fda53cf26b4f9ca63f6fbf9dc4c91b with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250115 02:52:18.780493 3504 catalog_manager.cc:5526] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d reported cstate change: config changed from index -1 to 2, VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) evicted. New cstate: current_term: 1 leader_uuid: "33878f6299994faf9451dc33aed46f4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } health_report { overall_health: HEALTHY } } }
W20250115 02:52:18.893401 3980 tablet_service.cc:374] Invalid argument: UpdateConsensus: Wrong destination UUID requested. Local UUID: 8cfd890708f24463b8f2c2e29e8eda79. Requested UUID: 027c4024e256416fb957e0045cea277e: from {username='slave'} at 127.0.0.1:49472: tablet_id: "bea6f3ae46a949f3a513871132a725c2" caller_uuid: "be1b14791ef64f6facefecabbae5f456" caller_term: 1 preceding_id { term: 1 index: 1 } dest_uuid: "027c4024e256416fb957e0045cea277e" committed_index: 1 all_replicated_index: 1 safe_timestamp: 7114381471050162176 last_idx_appended_to_leader: 1
I20250115 02:52:18.894842 3687 consensus_queue.cc:579] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [LEADER]: The tablet replica hosted on peer 027c4024e256416fb957e0045cea277e has failed
I20250115 02:52:18.914220 3722 consensus_queue.cc:237] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } }
I20250115 02:52:18.923631 3801 consensus_queue.cc:237] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: NON_VOTER last_known_addr { host: "127.2.36.65" port: 43753 } attrs { promote: true } }
W20250115 02:52:18.930094 3980 tablet_service.cc:374] Invalid argument: UpdateConsensus: Wrong destination UUID requested. Local UUID: 8cfd890708f24463b8f2c2e29e8eda79. Requested UUID: 027c4024e256416fb957e0045cea277e: from {username='slave'} at 127.0.0.1:49458: tablet_id: "59fda53cf26b4f9ca63f6fbf9dc4c91b" caller_uuid: "33878f6299994faf9451dc33aed46f4d" caller_term: 1 preceding_id { term: 1 index: 3 } dest_uuid: "027c4024e256416fb957e0045cea277e" committed_index: 2 all_replicated_index: 0 safe_timestamp: 7114381471446564864 last_idx_appended_to_leader: 3
I20250115 02:52:18.932484 3723 raft_consensus.cc:1270] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Refusing update from remote peer 33878f6299994faf9451dc33aed46f4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250115 02:52:18.939632 3909 consensus_queue.cc:1035] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250115 02:52:18.941735 3765 consensus_queue.cc:579] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [LEADER]: The tablet replica hosted on peer 027c4024e256416fb957e0045cea277e has failed
W20250115 02:52:18.942188 3765 consensus_peers.cc:487] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d -> Peer 027c4024e256416fb957e0045cea277e (127.2.36.65:43753): Couldn't send request to peer 027c4024e256416fb957e0045cea277e. Error code: WRONG_SERVER_UUID (16). Status: Invalid argument: UpdateConsensus: Wrong destination UUID requested. Local UUID: 8cfd890708f24463b8f2c2e29e8eda79. Requested UUID: 027c4024e256416fb957e0045cea277e. This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:18.953264 4018 raft_consensus.cc:2949] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: NON_VOTER last_known_addr { host: "127.2.36.65" port: 43753 } attrs { promote: true } } }
I20250115 02:52:18.958746 3723 raft_consensus.cc:2949] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: NON_VOTER last_known_addr { host: "127.2.36.65" port: 43753 } attrs { promote: true } } }
I20250115 02:52:18.972275 2193 tablet_server.cc:178] TabletServer@127.2.36.65:43753 shutting down...
I20250115 02:52:18.973601 3801 raft_consensus.cc:1270] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d [term 1 FOLLOWER]: Refusing update from remote peer be1b14791ef64f6facefecabbae5f456: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:18.983047 3875 consensus_queue.cc:1035] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20250115 02:52:18.995258 3492 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 59fda53cf26b4f9ca63f6fbf9dc4c91b with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250115 02:52:18.998394 3504 catalog_manager.cc:5526] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d reported cstate change: config changed from index 2 to 3, NON_VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) added. New cstate: current_term: 1 leader_uuid: "33878f6299994faf9451dc33aed46f4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: NON_VOTER last_known_addr { host: "127.2.36.65" port: 43753 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
W20250115 02:52:19.000368 3492 catalog_manager.cc:4615] TS 027c4024e256416fb957e0045cea277e (127.2.36.65:43753): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 59fda53cf26b4f9ca63f6fbf9dc4c91b: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
I20250115 02:52:19.017946 3875 raft_consensus.cc:2949] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } }
I20250115 02:52:19.027014 3801 raft_consensus.cc:2949] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } }
I20250115 02:52:19.039150 3493 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet bea6f3ae46a949f3a513871132a725c2 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250115 02:52:19.060230 3504 catalog_manager.cc:5526] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 reported cstate change: config changed from index -1 to 2, VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) evicted. New cstate: current_term: 1 leader_uuid: "be1b14791ef64f6facefecabbae5f456" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } health_report { overall_health: HEALTHY } } }
I20250115 02:52:19.061884 3801 consensus_queue.cc:237] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } }
I20250115 02:52:19.066372 3723 raft_consensus.cc:1270] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Refusing update from remote peer 33878f6299994faf9451dc33aed46f4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250115 02:52:19.068780 4019 consensus_queue.cc:1035] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250115 02:52:19.073035 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
W20250115 02:52:19.079876 3492 connection.cc:537] client connection to 127.2.36.65:43753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250115 02:52:19.080968 3909 raft_consensus.cc:2949] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } }
I20250115 02:52:19.083495 3723 consensus_queue.cc:237] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: NON_VOTER last_known_addr { host: "127.2.36.65" port: 43753 } attrs { promote: true } }
I20250115 02:52:19.088557 2193 tablet_server.cc:195] TabletServer@127.2.36.65:43753 shutdown complete.
I20250115 02:52:19.092460 3722 raft_consensus.cc:2949] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } }
W20250115 02:52:19.101935 3687 consensus_peers.cc:487] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 -> Peer 027c4024e256416fb957e0045cea277e (127.2.36.65:43753): Couldn't send request to peer 027c4024e256416fb957e0045cea277e. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:19.103246 3801 raft_consensus.cc:1270] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d [term 1 FOLLOWER]: Refusing update from remote peer be1b14791ef64f6facefecabbae5f456: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250115 02:52:19.103693 3506 catalog_manager.cc:5526] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d reported cstate change: config changed from index 3 to 4, NON_VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) evicted. New cstate: current_term: 1 leader_uuid: "33878f6299994faf9451dc33aed46f4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } health_report { overall_health: HEALTHY } } }
I20250115 02:52:19.106247 3492 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 59fda53cf26b4f9ca63f6fbf9dc4c91b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250115 02:52:19.106611 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:52:19.107573 3875 consensus_queue.cc:1035] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250115 02:52:19.139509 3801 consensus_queue.cc:237] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: NON_VOTER last_known_addr { host: "127.2.36.65" port: 43753 } attrs { promote: true } }
I20250115 02:52:19.141113 3947 raft_consensus.cc:2949] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: NON_VOTER last_known_addr { host: "127.2.36.65" port: 43753 } attrs { promote: true } } }
I20250115 02:52:19.155102 3493 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bea6f3ae46a949f3a513871132a725c2 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250115 02:52:19.154453 3800 raft_consensus.cc:2949] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: NON_VOTER last_known_addr { host: "127.2.36.65" port: 43753 } attrs { promote: true } } }
I20250115 02:52:19.169689 3506 catalog_manager.cc:5526] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 reported cstate change: config changed from index 2 to 3, NON_VOTER 027c4024e256416fb957e0045cea277e (127.2.36.65) added. New cstate: current_term: 1 leader_uuid: "be1b14791ef64f6facefecabbae5f456" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: NON_VOTER last_known_addr { host: "127.2.36.65" port: 43753 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250115 02:52:19.177564 3766 consensus_peers.cc:487] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d -> Peer be1b14791ef64f6facefecabbae5f456 (127.2.36.66:34895): Couldn't send request to peer be1b14791ef64f6facefecabbae5f456. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250115 02:52:19.187884 3765 consensus_peers.cc:487] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d -> Peer 027c4024e256416fb957e0045cea277e (127.2.36.65:43753): Couldn't send request to peer 027c4024e256416fb957e0045cea277e. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:19.196472 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:19.197165 2193 tablet_replica.cc:331] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456: stopping tablet replica
I20250115 02:52:19.197844 2193 raft_consensus.cc:2238] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:19.198204 2193 raft_consensus.cc:2267] T a9363d43a0ad4f559cc84450ba686b73 P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:19.199837 2193 tablet_replica.cc:331] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456: stopping tablet replica
I20250115 02:52:19.200297 2193 raft_consensus.cc:2238] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:19.201009 2193 raft_consensus.cc:2267] T bea6f3ae46a949f3a513871132a725c2 P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:19.203069 2193 tablet_replica.cc:331] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456: stopping tablet replica
I20250115 02:52:19.203792 2193 raft_consensus.cc:2238] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:19.204304 2193 raft_consensus.cc:2267] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P be1b14791ef64f6facefecabbae5f456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:19.227232 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:52:19.240262 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:52:19.266902 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:19.267411 2193 tablet_replica.cc:331] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d: stopping tablet replica
I20250115 02:52:19.267926 2193 raft_consensus.cc:2238] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:19.268333 2193 raft_consensus.cc:2267] T a9363d43a0ad4f559cc84450ba686b73 P 33878f6299994faf9451dc33aed46f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:19.270169 2193 tablet_replica.cc:331] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d: stopping tablet replica
I20250115 02:52:19.270903 2193 raft_consensus.cc:2238] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:19.271544 2193 raft_consensus.cc:2267] T bea6f3ae46a949f3a513871132a725c2 P 33878f6299994faf9451dc33aed46f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:19.273535 2193 tablet_replica.cc:331] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d: stopping tablet replica
I20250115 02:52:19.273967 2193 raft_consensus.cc:2238] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:19.274636 2193 pending_rounds.cc:62] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d: Trying to abort 1 pending ops.
I20250115 02:52:19.274922 2193 pending_rounds.cc:69] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7114381472311427072 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "59fda53cf26b4f9ca63f6fbf9dc4c91b" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be1b14791ef64f6facefecabbae5f456" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34895 } } peers { permanent_uuid: "33878f6299994faf9451dc33aed46f4d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41377 } } peers { permanent_uuid: "027c4024e256416fb957e0045cea277e" member_type: NON_VOTER last_known_addr { host: "127.2.36.65" port: 43753 } attrs { promote: true } } } }
I20250115 02:52:19.275710 2193 raft_consensus.cc:2911] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 1 LEADER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20250115 02:52:19.276069 2193 raft_consensus.cc:2883] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 1 LEADER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250115 02:52:19.276732 2193 raft_consensus.cc:2267] T 59fda53cf26b4f9ca63f6fbf9dc4c91b P 33878f6299994faf9451dc33aed46f4d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 02:52:19.277058 3492 catalog_manager.cc:5056] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 59fda53cf26b4f9ca63f6fbf9dc4c91b with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 33878f6299994faf9451dc33aed46f4d (127.2.36.67:41377) due to error UNKNOWN_ERROR; will retry: Aborted: Op aborted
I20250115 02:52:19.288955 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:52:19.300629 2193 master.cc:537] Master@127.2.36.126:43075 shutting down...
I20250115 02:52:19.348182 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:19.348730 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:19.349046 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 18efeacf16b84ef180b17a4496a6a46b: stopping tablet replica
I20250115 02:52:19.360596 2193 master.cc:559] Master@127.2.36.126:43075 shutdown complete.
I20250115 02:52:19.373752 2193 master.cc:537] Master@127.2.36.125:44051 shutting down...
I20250115 02:52:19.392642 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:19.393126 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:19.393431 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P a3d2c87272ae4171982243b0742b130d: stopping tablet replica
I20250115 02:52:19.412096 2193 master.cc:559] Master@127.2.36.125:44051 shutdown complete.
I20250115 02:52:19.422266 2193 master.cc:537] Master@127.2.36.124:46361 shutting down...
W20250115 02:52:19.594044 3493 consensus_peers.cc:487] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 -> Peer 18efeacf16b84ef180b17a4496a6a46b (127.2.36.126:43075): Couldn't send request to peer 18efeacf16b84ef180b17a4496a6a46b. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.126:43075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:19.645836 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:19.646817 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:19.647136 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 468df4ed46084aeea2cee0acdbad70a1: stopping tablet replica
I20250115 02:52:19.667320 2193 master.cc:559] Master@127.2.36.124:46361 shutdown complete.
[ OK ] RemoteKsckTest.TestTabletServerMismatchedUUID (4165 ms)
[ RUN ] RemoteKsckTest.TestTableConsistency
I20250115 02:52:19.697638 2193 test_util.cc:274] Using random seed: -1004608564
I20250115 02:52:19.698555 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:40401,127.2.36.125:37199,127.2.36.124:46587
I20250115 02:52:19.699810 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:19.705176 4039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:19.705771 4040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:19.706321 4042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:19.707250 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:19.708141 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:19.708308 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:19.708422 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909539708411 us; error 0 us; skew 500 ppm
I20250115 02:52:19.708859 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:19.711026 2193 webserver.cc:458] Webserver started at http://127.2.36.126:45047/ using document root <none> and password file <none>
I20250115 02:52:19.711436 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:19.711581 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:19.711773 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:19.712724 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "1f7fb9698c994356986f9b8af5dc9d78"
format_stamp: "Formatted at 2025-01-15 02:52:19 on dist-test-slave-1lv2"
I20250115 02:52:19.716907 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20250115 02:52:19.719828 4047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:19.720494 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:52:19.720752 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "1f7fb9698c994356986f9b8af5dc9d78"
format_stamp: "Formatted at 2025-01-15 02:52:19 on dist-test-slave-1lv2"
I20250115 02:52:19.721015 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:19.737895 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:19.739079 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:19.774194 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:40401
I20250115 02:52:19.774279 4098 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:40401 every 8 connection(s)
I20250115 02:52:19.777765 4099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:19.777910 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:19.783944 4101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:19.784775 4102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:19.784696 4099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:19.785902 4104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:19.787272 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:19.788379 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:19.788578 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:19.788734 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909539788717 us; error 0 us; skew 500 ppm
I20250115 02:52:19.789208 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:19.792380 2193 webserver.cc:458] Webserver started at http://127.2.36.125:37355/ using document root <none> and password file <none>
I20250115 02:52:19.792920 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:19.793148 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:19.793448 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:19.794737 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "2d9691daf7bc4374bbe106202f42cb4a"
format_stamp: "Formatted at 2025-01-15 02:52:19 on dist-test-slave-1lv2"
I20250115 02:52:19.797801 4099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:19.800148 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
W20250115 02:52:19.803225 4099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:37199: Network error: Client connection negotiation failed: client connection to 127.2.36.125:37199: connect: Connection refused (error 111)
I20250115 02:52:19.803903 4112 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:19.804627 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20250115 02:52:19.804903 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "2d9691daf7bc4374bbe106202f42cb4a"
format_stamp: "Formatted at 2025-01-15 02:52:19 on dist-test-slave-1lv2"
I20250115 02:52:19.805161 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:19.817687 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:19.818742 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:19.837769 4099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } attempt: 1
W20250115 02:52:19.842432 4099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:37199: Network error: Client connection negotiation failed: client connection to 127.2.36.125:37199: connect: Connection refused (error 111)
I20250115 02:52:19.855309 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:37199
I20250115 02:52:19.855399 4163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:37199 every 8 connection(s)
I20250115 02:52:19.858968 4164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:19.859053 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:19.865737 4164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:19.865999 4167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:19.866079 4166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:19.867033 4169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:19.867830 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:19.868711 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:19.868888 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:19.869010 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909539868999 us; error 0 us; skew 500 ppm
I20250115 02:52:19.869459 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:19.872319 2193 webserver.cc:458] Webserver started at http://127.2.36.124:34865/ using document root <none> and password file <none>
I20250115 02:52:19.872767 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:19.872938 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:19.873164 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:19.874146 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "1538993b13f44c1389830d21b5b27e05"
format_stamp: "Formatted at 2025-01-15 02:52:19 on dist-test-slave-1lv2"
I20250115 02:52:19.874974 4099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } attempt: 2
I20250115 02:52:19.878731 4164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:19.895210 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.021s user 0.012s sys 0.010s
I20250115 02:52:19.896371 4099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:19.900204 4177 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20250115 02:52:19.901124 4099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:46587: Network error: Client connection negotiation failed: client connection to 127.2.36.124:46587: connect: Connection refused (error 111)
I20250115 02:52:19.901376 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250115 02:52:19.901736 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "1538993b13f44c1389830d21b5b27e05"
format_stamp: "Formatted at 2025-01-15 02:52:19 on dist-test-slave-1lv2"
I20250115 02:52:19.901993 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:19.902149 4164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:19.912779 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20250115 02:52:19.913280 4164 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:46587: Network error: Client connection negotiation failed: client connection to 127.2.36.124:46587: connect: Connection refused (error 111)
I20250115 02:52:19.913930 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:19.949397 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:46587
I20250115 02:52:19.949497 4229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:46587 every 8 connection(s)
I20250115 02:52:19.952188 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:52:19.953092 4230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:19.955608 4099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } attempt: 1
I20250115 02:52:19.959740 4230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:19.970816 4230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:19.971810 4099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78: Bootstrap starting.
I20250115 02:52:19.976511 4099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:19.978786 4164 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } attempt: 1
I20250115 02:52:19.980930 4230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:19.981209 4099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78: No bootstrap required, opened a new log
I20250115 02:52:19.984133 4099 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } }
I20250115 02:52:19.984953 4099 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:19.985241 4099 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f7fb9698c994356986f9b8af5dc9d78, State: Initialized, Role: FOLLOWER
I20250115 02:52:19.986061 4099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } }
I20250115 02:52:19.988287 4236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } } }
I20250115 02:52:19.989074 4236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:19.990180 4099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:19.994177 4236 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:19.994735 4236 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } }
I20250115 02:52:19.995769 4164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a: Bootstrap starting.
I20250115 02:52:19.997543 4236 leader_election.cc:290] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d9691daf7bc4374bbe106202f42cb4a (127.2.36.125:37199), 1538993b13f44c1389830d21b5b27e05 (127.2.36.124:46587)
I20250115 02:52:20.003919 4164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:20.009148 4230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05: Bootstrap starting.
I20250115 02:52:20.015651 4230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:20.021800 4230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05: No bootstrap required, opened a new log
I20250115 02:52:20.022337 4164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a: No bootstrap required, opened a new log
I20250115 02:52:20.024796 4230 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } }
I20250115 02:52:20.024952 4164 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } }
I20250115 02:52:20.025615 4230 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:20.025674 4164 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:20.026086 4164 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9691daf7bc4374bbe106202f42cb4a, State: Initialized, Role: FOLLOWER
I20250115 02:52:20.026808 4164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } }
I20250115 02:52:20.028115 4230 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1538993b13f44c1389830d21b5b27e05, State: Initialized, Role: FOLLOWER
I20250115 02:52:20.028836 4230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } }
I20250115 02:52:20.029067 4248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } } }
I20250115 02:52:20.030112 4248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:20.032397 4164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:20.032480 4139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f7fb9698c994356986f9b8af5dc9d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9691daf7bc4374bbe106202f42cb4a" is_pre_election: true
I20250115 02:52:20.032974 4230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:20.033349 4205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f7fb9698c994356986f9b8af5dc9d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1538993b13f44c1389830d21b5b27e05" is_pre_election: true
I20250115 02:52:20.034031 4205 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f7fb9698c994356986f9b8af5dc9d78 in term 0.
I20250115 02:52:20.035530 4048 leader_election.cc:304] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1538993b13f44c1389830d21b5b27e05, 1f7fb9698c994356986f9b8af5dc9d78; no voters:
I20250115 02:52:20.036455 4236 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:20.036813 4236 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:20.037115 4236 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:20.043205 4139 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f7fb9698c994356986f9b8af5dc9d78 in term 0.
I20250115 02:52:20.045850 4236 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } }
I20250115 02:52:20.048952 4139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f7fb9698c994356986f9b8af5dc9d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9691daf7bc4374bbe106202f42cb4a"
I20250115 02:52:20.049623 4139 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:20.056852 4139 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7fb9698c994356986f9b8af5dc9d78 in term 1.
I20250115 02:52:20.058208 4048 leader_election.cc:304] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f7fb9698c994356986f9b8af5dc9d78, 2d9691daf7bc4374bbe106202f42cb4a; no voters:
I20250115 02:52:20.032343 4249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } } }
I20250115 02:52:20.062547 4249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:20.064237 4236 leader_election.cc:290] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [CANDIDATE]: Term 1 election: Requested vote from peers 2d9691daf7bc4374bbe106202f42cb4a (127.2.36.125:37199), 1538993b13f44c1389830d21b5b27e05 (127.2.36.124:46587)
I20250115 02:52:20.064893 4236 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:20.065322 4236 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 1 LEADER]: Becoming Leader. State: Replica: 1f7fb9698c994356986f9b8af5dc9d78, State: Running, Role: LEADER
I20250115 02:52:20.070684 4205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f7fb9698c994356986f9b8af5dc9d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1538993b13f44c1389830d21b5b27e05"
I20250115 02:52:20.071367 4205 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:20.079502 4205 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7fb9698c994356986f9b8af5dc9d78 in term 1.
I20250115 02:52:20.066159 4236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } }
I20250115 02:52:20.087311 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20250115 02:52:20.096822 4258 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:20.097263 4258 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:20.098953 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:52:20.099146 4260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f7fb9698c994356986f9b8af5dc9d78. Latest consensus state: current_term: 1 leader_uuid: "1f7fb9698c994356986f9b8af5dc9d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } } }
I20250115 02:52:20.100052 4260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:20.101878 4272 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
W20250115 02:52:20.102835 4271 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:20.103332 4271 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:20.107412 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:20.108285 4272 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250115 02:52:20.108832 4276 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:20.109090 4276 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:52:20.115528 4277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:20.116556 4278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:20.119647 4205 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [term 1 FOLLOWER]: Refusing update from remote peer 1f7fb9698c994356986f9b8af5dc9d78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:20.120720 4139 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [term 1 FOLLOWER]: Refusing update from remote peer 1f7fb9698c994356986f9b8af5dc9d78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:20.122181 4260 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:20.123356 4236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:20.127909 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:20.130067 4249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f7fb9698c994356986f9b8af5dc9d78. Latest consensus state: current_term: 1 leader_uuid: "1f7fb9698c994356986f9b8af5dc9d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } } }
I20250115 02:52:20.130616 4249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [sys.catalog]: This master's current role is: FOLLOWER
W20250115 02:52:20.132557 4281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:20.133723 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:20.134011 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:20.134207 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909540134191 us; error 0 us; skew 500 ppm
I20250115 02:52:20.134917 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:20.135382 4260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f7fb9698c994356986f9b8af5dc9d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } } }
I20250115 02:52:20.136114 4260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:20.137266 4272 catalog_manager.cc:1348] Generated new cluster ID: fc051068fc3a416694f185bc4e2576a9
I20250115 02:52:20.137641 4272 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:52:20.143369 4248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f7fb9698c994356986f9b8af5dc9d78. Latest consensus state: current_term: 1 leader_uuid: "1f7fb9698c994356986f9b8af5dc9d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } } }
I20250115 02:52:20.144158 4248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:20.145721 4249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f7fb9698c994356986f9b8af5dc9d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } } }
I20250115 02:52:20.146458 4249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:20.146968 4236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f7fb9698c994356986f9b8af5dc9d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } } }
I20250115 02:52:20.147701 4236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:20.149878 2193 webserver.cc:458] Webserver started at http://127.2.36.65:38233/ using document root <none> and password file <none>
I20250115 02:52:20.150564 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:20.150837 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:20.151165 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:20.152653 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "4b9c97e21437419383aed90eca8a4ffa"
format_stamp: "Formatted at 2025-01-15 02:52:20 on dist-test-slave-1lv2"
I20250115 02:52:20.159396 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250115 02:52:20.162552 4248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f7fb9698c994356986f9b8af5dc9d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7fb9698c994356986f9b8af5dc9d78" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40401 } } peers { permanent_uuid: "2d9691daf7bc4374bbe106202f42cb4a" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 37199 } } peers { permanent_uuid: "1538993b13f44c1389830d21b5b27e05" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 46587 } } }
I20250115 02:52:20.163345 4248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:20.165143 4294 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:20.165961 2193 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20250115 02:52:20.166236 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "4b9c97e21437419383aed90eca8a4ffa"
format_stamp: "Formatted at 2025-01-15 02:52:20 on dist-test-slave-1lv2"
I20250115 02:52:20.166504 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:20.177728 4272 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:52:20.179461 4272 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:52:20.205680 4272 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78: Generated new TSK 0
I20250115 02:52:20.206274 4272 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:52:20.210172 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:20.211292 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:20.212656 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:20.214869 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:20.215063 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:20.215279 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:20.215430 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:20.252272 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:45083
I20250115 02:52:20.252372 4356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:45083 every 8 connection(s)
I20250115 02:52:20.259048 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:20.273721 4358 heartbeater.cc:346] Connected to a master server at 127.2.36.125:37199
I20250115 02:52:20.273850 4357 heartbeater.cc:346] Connected to a master server at 127.2.36.124:46587
I20250115 02:52:20.274243 4358 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:20.274286 4357 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:20.275242 4358 heartbeater.cc:509] Master 127.2.36.125:37199 requested a full tablet report, sending...
I20250115 02:52:20.275885 4357 heartbeater.cc:509] Master 127.2.36.124:46587 requested a full tablet report, sending...
W20250115 02:52:20.276346 4364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:20.278776 4365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:20.278892 4129 ts_manager.cc:194] Registered new tserver with Master: 4b9c97e21437419383aed90eca8a4ffa (127.2.36.65:45083)
I20250115 02:52:20.282040 4195 ts_manager.cc:194] Registered new tserver with Master: 4b9c97e21437419383aed90eca8a4ffa (127.2.36.65:45083)
I20250115 02:52:20.282562 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:20.283025 4359 heartbeater.cc:346] Connected to a master server at 127.2.36.126:40401
W20250115 02:52:20.283281 4367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:20.283398 4359 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:20.284205 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250115 02:52:20.284296 4359 heartbeater.cc:509] Master 127.2.36.126:40401 requested a full tablet report, sending...
W20250115 02:52:20.284574 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:20.284806 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909540284791 us; error 0 us; skew 500 ppm
I20250115 02:52:20.285362 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:20.286685 4064 ts_manager.cc:194] Registered new tserver with Master: 4b9c97e21437419383aed90eca8a4ffa (127.2.36.65:45083)
I20250115 02:52:20.288177 2193 webserver.cc:458] Webserver started at http://127.2.36.66:42369/ using document root <none> and password file <none>
I20250115 02:52:20.288820 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:20.289050 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:20.289013 4064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34394
I20250115 02:52:20.289527 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:20.290828 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "d93156ddb0f2457fa3ba5b43150885ae"
format_stamp: "Formatted at 2025-01-15 02:52:20 on dist-test-slave-1lv2"
I20250115 02:52:20.296254 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20250115 02:52:20.299427 4372 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:20.300141 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:52:20.300419 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "d93156ddb0f2457fa3ba5b43150885ae"
format_stamp: "Formatted at 2025-01-15 02:52:20 on dist-test-slave-1lv2"
I20250115 02:52:20.300678 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:20.301582 4359 heartbeater.cc:501] Master 127.2.36.126:40401 was elected leader, sending a full tablet report...
I20250115 02:52:20.331372 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:20.332890 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:20.334575 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:20.336876 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:20.337076 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:20.337293 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:20.337443 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:20.380708 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:43333
I20250115 02:52:20.380784 4434 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:43333 every 8 connection(s)
I20250115 02:52:20.410923 4435 heartbeater.cc:346] Connected to a master server at 127.2.36.124:46587
I20250115 02:52:20.411079 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:20.411319 4435 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:20.412149 4435 heartbeater.cc:509] Master 127.2.36.124:46587 requested a full tablet report, sending...
I20250115 02:52:20.413154 4436 heartbeater.cc:346] Connected to a master server at 127.2.36.125:37199
I20250115 02:52:20.413511 4436 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:20.414285 4436 heartbeater.cc:509] Master 127.2.36.125:37199 requested a full tablet report, sending...
I20250115 02:52:20.417357 4194 ts_manager.cc:194] Registered new tserver with Master: d93156ddb0f2457fa3ba5b43150885ae (127.2.36.66:43333)
W20250115 02:52:20.420264 4443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:20.420348 4129 ts_manager.cc:194] Registered new tserver with Master: d93156ddb0f2457fa3ba5b43150885ae (127.2.36.66:43333)
I20250115 02:52:20.420564 4437 heartbeater.cc:346] Connected to a master server at 127.2.36.126:40401
I20250115 02:52:20.421083 4437 heartbeater.cc:463] Registering TS with master...
W20250115 02:52:20.421743 4442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:20.422039 4437 heartbeater.cc:509] Master 127.2.36.126:40401 requested a full tablet report, sending...
I20250115 02:52:20.424388 4064 ts_manager.cc:194] Registered new tserver with Master: d93156ddb0f2457fa3ba5b43150885ae (127.2.36.66:43333)
I20250115 02:52:20.425657 4064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34402
I20250115 02:52:20.427042 2193 server_base.cc:1034] running on GCE node
W20250115 02:52:20.428714 4445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:20.429842 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:20.430119 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:20.430342 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909540430325 us; error 0 us; skew 500 ppm
I20250115 02:52:20.430986 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:20.433562 2193 webserver.cc:458] Webserver started at http://127.2.36.67:35753/ using document root <none> and password file <none>
I20250115 02:52:20.434162 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:20.434412 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:20.434681 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:20.435966 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "a34437bd4f3b47368d99913d56ecef7d"
format_stamp: "Formatted at 2025-01-15 02:52:20 on dist-test-slave-1lv2"
I20250115 02:52:20.438593 4437 heartbeater.cc:501] Master 127.2.36.126:40401 was elected leader, sending a full tablet report...
I20250115 02:52:20.440357 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:52:20.445758 4450 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:20.446521 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20250115 02:52:20.446844 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "a34437bd4f3b47368d99913d56ecef7d"
format_stamp: "Formatted at 2025-01-15 02:52:20 on dist-test-slave-1lv2"
I20250115 02:52:20.447098 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableConsistency.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:20.471011 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:20.472481 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:20.474308 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:20.476706 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:20.476894 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:20.477125 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:20.477277 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:20.540007 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:39029
I20250115 02:52:20.540100 4512 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:39029 every 8 connection(s)
I20250115 02:52:20.563650 4513 heartbeater.cc:346] Connected to a master server at 127.2.36.124:46587
I20250115 02:52:20.564038 4513 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:20.564884 4513 heartbeater.cc:509] Master 127.2.36.124:46587 requested a full tablet report, sending...
I20250115 02:52:20.567024 4514 heartbeater.cc:346] Connected to a master server at 127.2.36.125:37199
I20250115 02:52:20.567407 4514 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:20.567845 4194 ts_manager.cc:194] Registered new tserver with Master: a34437bd4f3b47368d99913d56ecef7d (127.2.36.67:39029)
I20250115 02:52:20.568288 4514 heartbeater.cc:509] Master 127.2.36.125:37199 requested a full tablet report, sending...
I20250115 02:52:20.570622 4515 heartbeater.cc:346] Connected to a master server at 127.2.36.126:40401
I20250115 02:52:20.570880 4129 ts_manager.cc:194] Registered new tserver with Master: a34437bd4f3b47368d99913d56ecef7d (127.2.36.67:39029)
I20250115 02:52:20.571000 4515 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:20.571897 4515 heartbeater.cc:509] Master 127.2.36.126:40401 requested a full tablet report, sending...
I20250115 02:52:20.573632 4064 ts_manager.cc:194] Registered new tserver with Master: a34437bd4f3b47368d99913d56ecef7d (127.2.36.67:39029)
I20250115 02:52:20.574326 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.028155536s
I20250115 02:52:20.575198 4064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34404
I20250115 02:52:20.592391 4515 heartbeater.cc:501] Master 127.2.36.126:40401 was elected leader, sending a full tablet report...
I20250115 02:52:20.627982 4064 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34408:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:52:20.631104 4064 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:52:20.730165 4322 tablet_service.cc:1467] Processing CreateTablet for tablet a40ace077e2d43e8bfc88d20a8bb9920 (DEFAULT_TABLE table=ksck-test-table [id=67e4f0075cc04ea5bd401606043a972a]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:20.731272 4322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40ace077e2d43e8bfc88d20a8bb9920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:20.737484 4321 tablet_service.cc:1467] Processing CreateTablet for tablet d7a566fcd649414e86cfda163a4f1eb4 (DEFAULT_TABLE table=ksck-test-table [id=67e4f0075cc04ea5bd401606043a972a]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:20.738875 4321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7a566fcd649414e86cfda163a4f1eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:20.739699 4320 tablet_service.cc:1467] Processing CreateTablet for tablet 65a92db1130849ad9aa850dc4f5c87d8 (DEFAULT_TABLE table=ksck-test-table [id=67e4f0075cc04ea5bd401606043a972a]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:20.740935 4320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65a92db1130849ad9aa850dc4f5c87d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:20.762516 4535 tablet_bootstrap.cc:492] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa: Bootstrap starting.
I20250115 02:52:20.774689 4535 tablet_bootstrap.cc:654] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:20.785043 4535 tablet_bootstrap.cc:492] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa: No bootstrap required, opened a new log
I20250115 02:52:20.785475 4535 ts_tablet_manager.cc:1397] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.009s
I20250115 02:52:20.788005 4535 raft_consensus.cc:357] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:20.788570 4535 raft_consensus.cc:383] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:20.788808 4535 raft_consensus.cc:738] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9c97e21437419383aed90eca8a4ffa, State: Initialized, Role: FOLLOWER
I20250115 02:52:20.789357 4535 consensus_queue.cc:260] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:20.791910 4478 tablet_service.cc:1467] Processing CreateTablet for tablet a40ace077e2d43e8bfc88d20a8bb9920 (DEFAULT_TABLE table=ksck-test-table [id=67e4f0075cc04ea5bd401606043a972a]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:20.793782 4535 ts_tablet_manager.cc:1428] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20250115 02:52:20.794615 4476 tablet_service.cc:1467] Processing CreateTablet for tablet 65a92db1130849ad9aa850dc4f5c87d8 (DEFAULT_TABLE table=ksck-test-table [id=67e4f0075cc04ea5bd401606043a972a]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:20.795944 4476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65a92db1130849ad9aa850dc4f5c87d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:20.799399 4477 tablet_service.cc:1467] Processing CreateTablet for tablet d7a566fcd649414e86cfda163a4f1eb4 (DEFAULT_TABLE table=ksck-test-table [id=67e4f0075cc04ea5bd401606043a972a]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:20.800696 4477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7a566fcd649414e86cfda163a4f1eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:20.807157 4400 tablet_service.cc:1467] Processing CreateTablet for tablet a40ace077e2d43e8bfc88d20a8bb9920 (DEFAULT_TABLE table=ksck-test-table [id=67e4f0075cc04ea5bd401606043a972a]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:20.807466 4398 tablet_service.cc:1467] Processing CreateTablet for tablet 65a92db1130849ad9aa850dc4f5c87d8 (DEFAULT_TABLE table=ksck-test-table [id=67e4f0075cc04ea5bd401606043a972a]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:20.808442 4535 tablet_bootstrap.cc:492] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa: Bootstrap starting.
I20250115 02:52:20.808522 4400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40ace077e2d43e8bfc88d20a8bb9920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:20.808912 4478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40ace077e2d43e8bfc88d20a8bb9920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:20.814611 4535 tablet_bootstrap.cc:654] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:20.814925 4399 tablet_service.cc:1467] Processing CreateTablet for tablet d7a566fcd649414e86cfda163a4f1eb4 (DEFAULT_TABLE table=ksck-test-table [id=67e4f0075cc04ea5bd401606043a972a]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:20.816298 4399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7a566fcd649414e86cfda163a4f1eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:20.822489 4398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65a92db1130849ad9aa850dc4f5c87d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:20.835798 4540 raft_consensus.cc:491] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:20.836318 4540 raft_consensus.cc:513] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:20.871006 4540 leader_election.cc:290] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d93156ddb0f2457fa3ba5b43150885ae (127.2.36.66:43333), a34437bd4f3b47368d99913d56ecef7d (127.2.36.67:39029)
I20250115 02:52:20.871762 4539 tablet_bootstrap.cc:492] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae: Bootstrap starting.
I20250115 02:52:20.886140 4535 tablet_bootstrap.cc:492] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa: No bootstrap required, opened a new log
I20250115 02:52:20.886716 4535 ts_tablet_manager.cc:1397] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.018s
I20250115 02:52:20.889446 4535 raft_consensus.cc:357] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } } peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } }
I20250115 02:52:20.890187 4535 raft_consensus.cc:383] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:20.890497 4535 raft_consensus.cc:738] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9c97e21437419383aed90eca8a4ffa, State: Initialized, Role: FOLLOWER
I20250115 02:52:20.890970 4539 tablet_bootstrap.cc:654] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:20.891280 4535 consensus_queue.cc:260] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } } peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } }
I20250115 02:52:20.895828 4535 ts_tablet_manager.cc:1428] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa: Time spent starting tablet: real 0.009s user 0.000s sys 0.007s
I20250115 02:52:20.896950 4535 tablet_bootstrap.cc:492] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa: Bootstrap starting.
I20250115 02:52:20.905529 4535 tablet_bootstrap.cc:654] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:20.923674 4488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a566fcd649414e86cfda163a4f1eb4" candidate_uuid: "4b9c97e21437419383aed90eca8a4ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a34437bd4f3b47368d99913d56ecef7d" is_pre_election: true
I20250115 02:52:20.929463 4410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a566fcd649414e86cfda163a4f1eb4" candidate_uuid: "4b9c97e21437419383aed90eca8a4ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93156ddb0f2457fa3ba5b43150885ae" is_pre_election: true
W20250115 02:52:20.929167 4298 leader_election.cc:343] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a34437bd4f3b47368d99913d56ecef7d (127.2.36.67:39029): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:52:20.931950 4545 tablet_bootstrap.cc:492] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d: Bootstrap starting.
I20250115 02:52:20.932483 4535 tablet_bootstrap.cc:492] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa: No bootstrap required, opened a new log
I20250115 02:52:20.932968 4535 ts_tablet_manager.cc:1397] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.006s
W20250115 02:52:20.936573 4295 leader_election.cc:343] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d93156ddb0f2457fa3ba5b43150885ae (127.2.36.66:43333): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:52:20.936825 4535 raft_consensus.cc:357] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:20.937611 4545 tablet_bootstrap.cc:654] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:20.937587 4535 raft_consensus.cc:383] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:20.938644 4295 leader_election.cc:304] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9c97e21437419383aed90eca8a4ffa; no voters: a34437bd4f3b47368d99913d56ecef7d, d93156ddb0f2457fa3ba5b43150885ae
I20250115 02:52:20.939474 4540 raft_consensus.cc:2743] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:52:20.941133 4539 tablet_bootstrap.cc:492] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae: No bootstrap required, opened a new log
I20250115 02:52:20.941609 4539 ts_tablet_manager.cc:1397] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae: Time spent bootstrapping tablet: real 0.070s user 0.008s sys 0.008s
I20250115 02:52:20.943799 4539 raft_consensus.cc:357] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } } peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } }
I20250115 02:52:20.944394 4539 raft_consensus.cc:383] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:20.944682 4539 raft_consensus.cc:738] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93156ddb0f2457fa3ba5b43150885ae, State: Initialized, Role: FOLLOWER
I20250115 02:52:20.945341 4539 consensus_queue.cc:260] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } } peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } }
I20250115 02:52:20.938170 4535 raft_consensus.cc:738] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9c97e21437419383aed90eca8a4ffa, State: Initialized, Role: FOLLOWER
I20250115 02:52:20.947363 4535 consensus_queue.cc:260] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:20.949712 4535 ts_tablet_manager.cc:1428] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa: Time spent starting tablet: real 0.016s user 0.005s sys 0.002s
I20250115 02:52:20.953027 4539 ts_tablet_manager.cc:1428] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250115 02:52:20.954115 4539 tablet_bootstrap.cc:492] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae: Bootstrap starting.
I20250115 02:52:20.961211 4539 tablet_bootstrap.cc:654] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:20.967093 4545 tablet_bootstrap.cc:492] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d: No bootstrap required, opened a new log
I20250115 02:52:20.967594 4545 ts_tablet_manager.cc:1397] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d: Time spent bootstrapping tablet: real 0.036s user 0.025s sys 0.008s
I20250115 02:52:20.972298 4545 raft_consensus.cc:357] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:20.973027 4545 raft_consensus.cc:383] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:20.973312 4545 raft_consensus.cc:738] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34437bd4f3b47368d99913d56ecef7d, State: Initialized, Role: FOLLOWER
I20250115 02:52:20.974231 4545 consensus_queue.cc:260] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:20.990487 4545 ts_tablet_manager.cc:1428] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d: Time spent starting tablet: real 0.023s user 0.011s sys 0.008s
I20250115 02:52:20.993645 4545 tablet_bootstrap.cc:492] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d: Bootstrap starting.
I20250115 02:52:21.005059 4545 tablet_bootstrap.cc:654] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:21.017524 4539 tablet_bootstrap.cc:492] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae: No bootstrap required, opened a new log
I20250115 02:52:21.017961 4539 ts_tablet_manager.cc:1397] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae: Time spent bootstrapping tablet: real 0.064s user 0.033s sys 0.026s
I20250115 02:52:21.019342 4545 tablet_bootstrap.cc:492] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d: No bootstrap required, opened a new log
I20250115 02:52:21.019856 4545 ts_tablet_manager.cc:1397] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.002s
I20250115 02:52:21.019992 4539 raft_consensus.cc:357] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:21.020675 4539 raft_consensus.cc:383] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:21.020957 4539 raft_consensus.cc:738] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93156ddb0f2457fa3ba5b43150885ae, State: Initialized, Role: FOLLOWER
I20250115 02:52:21.021597 4539 consensus_queue.cc:260] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:21.022449 4545 raft_consensus.cc:357] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:21.023236 4545 raft_consensus.cc:383] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:21.023527 4545 raft_consensus.cc:738] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34437bd4f3b47368d99913d56ecef7d, State: Initialized, Role: FOLLOWER
I20250115 02:52:21.024295 4545 consensus_queue.cc:260] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:21.028716 4545 ts_tablet_manager.cc:1428] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20250115 02:52:21.029960 4545 tablet_bootstrap.cc:492] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d: Bootstrap starting.
I20250115 02:52:21.033449 4539 ts_tablet_manager.cc:1428] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae: Time spent starting tablet: real 0.015s user 0.009s sys 0.001s
I20250115 02:52:21.034703 4539 tablet_bootstrap.cc:492] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae: Bootstrap starting.
I20250115 02:52:21.035740 4545 tablet_bootstrap.cc:654] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:21.040387 4539 tablet_bootstrap.cc:654] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:21.040881 4545 tablet_bootstrap.cc:492] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d: No bootstrap required, opened a new log
I20250115 02:52:21.041388 4545 ts_tablet_manager.cc:1397] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20250115 02:52:21.046692 4545 raft_consensus.cc:357] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } } peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } }
I20250115 02:52:21.052309 4545 raft_consensus.cc:383] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:21.052625 4545 raft_consensus.cc:738] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34437bd4f3b47368d99913d56ecef7d, State: Initialized, Role: FOLLOWER
I20250115 02:52:21.047598 4540 raft_consensus.cc:491] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:21.053287 4545 consensus_queue.cc:260] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } } peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } }
I20250115 02:52:21.053632 4540 raft_consensus.cc:513] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } } peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } }
I20250115 02:52:21.055528 4540 leader_election.cc:290] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a34437bd4f3b47368d99913d56ecef7d (127.2.36.67:39029), d93156ddb0f2457fa3ba5b43150885ae (127.2.36.66:43333)
I20250115 02:52:21.055709 4545 ts_tablet_manager.cc:1428] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d: Time spent starting tablet: real 0.014s user 0.007s sys 0.001s
I20250115 02:52:21.057009 4488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40ace077e2d43e8bfc88d20a8bb9920" candidate_uuid: "4b9c97e21437419383aed90eca8a4ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a34437bd4f3b47368d99913d56ecef7d" is_pre_election: true
I20250115 02:52:21.057658 4488 raft_consensus.cc:2463] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b9c97e21437419383aed90eca8a4ffa in term 0.
I20250115 02:52:21.058996 4410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40ace077e2d43e8bfc88d20a8bb9920" candidate_uuid: "4b9c97e21437419383aed90eca8a4ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93156ddb0f2457fa3ba5b43150885ae" is_pre_election: true
I20250115 02:52:21.059636 4410 raft_consensus.cc:2463] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b9c97e21437419383aed90eca8a4ffa in term 0.
I20250115 02:52:21.051126 4539 tablet_bootstrap.cc:492] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae: No bootstrap required, opened a new log
I20250115 02:52:21.060757 4295 leader_election.cc:304] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b9c97e21437419383aed90eca8a4ffa, d93156ddb0f2457fa3ba5b43150885ae; no voters:
I20250115 02:52:21.061246 4539 ts_tablet_manager.cc:1397] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.002s
I20250115 02:52:21.061770 4540 raft_consensus.cc:2798] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:21.062140 4540 raft_consensus.cc:491] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:21.062553 4540 raft_consensus.cc:3054] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:21.065030 4539 raft_consensus.cc:357] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:21.065599 4539 raft_consensus.cc:383] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:21.065829 4539 raft_consensus.cc:738] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93156ddb0f2457fa3ba5b43150885ae, State: Initialized, Role: FOLLOWER
I20250115 02:52:21.066516 4539 consensus_queue.cc:260] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:21.068430 4539 ts_tablet_manager.cc:1428] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250115 02:52:21.077565 4540 raft_consensus.cc:513] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } } peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } }
I20250115 02:52:21.079681 4540 leader_election.cc:290] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [CANDIDATE]: Term 1 election: Requested vote from peers a34437bd4f3b47368d99913d56ecef7d (127.2.36.67:39029), d93156ddb0f2457fa3ba5b43150885ae (127.2.36.66:43333)
I20250115 02:52:21.080453 4488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40ace077e2d43e8bfc88d20a8bb9920" candidate_uuid: "4b9c97e21437419383aed90eca8a4ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a34437bd4f3b47368d99913d56ecef7d"
I20250115 02:52:21.080758 4410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40ace077e2d43e8bfc88d20a8bb9920" candidate_uuid: "4b9c97e21437419383aed90eca8a4ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93156ddb0f2457fa3ba5b43150885ae"
I20250115 02:52:21.081311 4410 raft_consensus.cc:3054] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:21.082901 4488 raft_consensus.cc:3054] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:21.091363 4410 raft_consensus.cc:2463] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9c97e21437419383aed90eca8a4ffa in term 1.
I20250115 02:52:21.092573 4295 leader_election.cc:304] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b9c97e21437419383aed90eca8a4ffa, d93156ddb0f2457fa3ba5b43150885ae; no voters:
I20250115 02:52:21.094496 4540 raft_consensus.cc:2798] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:21.095036 4540 raft_consensus.cc:695] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 1 LEADER]: Becoming Leader. State: Replica: 4b9c97e21437419383aed90eca8a4ffa, State: Running, Role: LEADER
I20250115 02:52:21.095824 4540 consensus_queue.cc:237] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } } peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } }
I20250115 02:52:21.096486 4488 raft_consensus.cc:2463] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9c97e21437419383aed90eca8a4ffa in term 1.
I20250115 02:52:21.108323 4271 catalog_manager.cc:1260] Loaded cluster ID: fc051068fc3a416694f185bc4e2576a9
I20250115 02:52:21.108604 4271 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a: loading cluster ID for follower catalog manager: success
I20250115 02:52:21.112526 4062 catalog_manager.cc:5526] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b9c97e21437419383aed90eca8a4ffa (127.2.36.65). New cstate: current_term: 1 leader_uuid: "4b9c97e21437419383aed90eca8a4ffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:21.114076 4271 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a: acquiring CA information for follower catalog manager: success
I20250115 02:52:21.114475 4276 catalog_manager.cc:1260] Loaded cluster ID: fc051068fc3a416694f185bc4e2576a9
I20250115 02:52:21.114794 4276 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05: loading cluster ID for follower catalog manager: success
I20250115 02:52:21.141494 4271 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:21.141821 4276 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05: acquiring CA information for follower catalog manager: success
I20250115 02:52:21.148043 4276 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:21.171053 4547 raft_consensus.cc:491] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:21.171612 4547 raft_consensus.cc:513] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:21.172228 4540 raft_consensus.cc:491] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:21.172755 4540 raft_consensus.cc:513] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:21.173744 4547 leader_election.cc:290] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b9c97e21437419383aed90eca8a4ffa (127.2.36.65:45083), a34437bd4f3b47368d99913d56ecef7d (127.2.36.67:39029)
I20250115 02:52:21.174654 4540 leader_election.cc:290] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d93156ddb0f2457fa3ba5b43150885ae (127.2.36.66:43333), a34437bd4f3b47368d99913d56ecef7d (127.2.36.67:39029)
I20250115 02:52:21.175377 4410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a92db1130849ad9aa850dc4f5c87d8" candidate_uuid: "4b9c97e21437419383aed90eca8a4ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93156ddb0f2457fa3ba5b43150885ae" is_pre_election: true
I20250115 02:52:21.180274 4410 raft_consensus.cc:2463] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b9c97e21437419383aed90eca8a4ffa in term 0.
I20250115 02:52:21.181932 4295 leader_election.cc:304] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b9c97e21437419383aed90eca8a4ffa, d93156ddb0f2457fa3ba5b43150885ae; no voters:
I20250115 02:52:21.184011 4540 raft_consensus.cc:2798] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:21.184376 4540 raft_consensus.cc:491] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:21.184715 4540 raft_consensus.cc:3054] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:21.178107 4488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a92db1130849ad9aa850dc4f5c87d8" candidate_uuid: "4b9c97e21437419383aed90eca8a4ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a34437bd4f3b47368d99913d56ecef7d" is_pre_election: true
I20250115 02:52:21.188628 4488 raft_consensus.cc:2463] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b9c97e21437419383aed90eca8a4ffa in term 0.
I20250115 02:52:21.191743 4540 raft_consensus.cc:513] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:21.193989 4540 leader_election.cc:290] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [CANDIDATE]: Term 1 election: Requested vote from peers d93156ddb0f2457fa3ba5b43150885ae (127.2.36.66:43333), a34437bd4f3b47368d99913d56ecef7d (127.2.36.67:39029)
I20250115 02:52:21.195048 4488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a92db1130849ad9aa850dc4f5c87d8" candidate_uuid: "4b9c97e21437419383aed90eca8a4ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a34437bd4f3b47368d99913d56ecef7d"
I20250115 02:52:21.196005 4410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a92db1130849ad9aa850dc4f5c87d8" candidate_uuid: "4b9c97e21437419383aed90eca8a4ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93156ddb0f2457fa3ba5b43150885ae"
I20250115 02:52:21.198349 4487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a566fcd649414e86cfda163a4f1eb4" candidate_uuid: "d93156ddb0f2457fa3ba5b43150885ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a34437bd4f3b47368d99913d56ecef7d" is_pre_election: true
I20250115 02:52:21.205925 4410 raft_consensus.cc:3054] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:21.206399 4487 raft_consensus.cc:2463] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d93156ddb0f2457fa3ba5b43150885ae in term 0.
I20250115 02:52:21.207389 4376 leader_election.cc:304] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a34437bd4f3b47368d99913d56ecef7d, d93156ddb0f2457fa3ba5b43150885ae; no voters:
I20250115 02:52:21.208212 4547 raft_consensus.cc:2798] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:21.208577 4547 raft_consensus.cc:491] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:21.208721 4488 raft_consensus.cc:3054] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:21.208976 4547 raft_consensus.cc:3054] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:21.212512 4410 raft_consensus.cc:2463] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9c97e21437419383aed90eca8a4ffa in term 1.
I20250115 02:52:21.213559 4295 leader_election.cc:304] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b9c97e21437419383aed90eca8a4ffa, d93156ddb0f2457fa3ba5b43150885ae; no voters:
I20250115 02:52:21.214316 4537 raft_consensus.cc:2798] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:21.223484 4547 raft_consensus.cc:513] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:21.226438 4487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a566fcd649414e86cfda163a4f1eb4" candidate_uuid: "d93156ddb0f2457fa3ba5b43150885ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a34437bd4f3b47368d99913d56ecef7d"
I20250115 02:52:21.227120 4487 raft_consensus.cc:3054] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:21.230466 4332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a566fcd649414e86cfda163a4f1eb4" candidate_uuid: "d93156ddb0f2457fa3ba5b43150885ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b9c97e21437419383aed90eca8a4ffa" is_pre_election: true
I20250115 02:52:21.232425 4332 raft_consensus.cc:2463] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d93156ddb0f2457fa3ba5b43150885ae in term 0.
I20250115 02:52:21.226824 4488 raft_consensus.cc:2463] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9c97e21437419383aed90eca8a4ffa in term 1.
I20250115 02:52:21.235415 4547 leader_election.cc:290] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [CANDIDATE]: Term 1 election: Requested vote from peers 4b9c97e21437419383aed90eca8a4ffa (127.2.36.65:45083), a34437bd4f3b47368d99913d56ecef7d (127.2.36.67:39029)
I20250115 02:52:21.235738 4332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a566fcd649414e86cfda163a4f1eb4" candidate_uuid: "d93156ddb0f2457fa3ba5b43150885ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b9c97e21437419383aed90eca8a4ffa"
I20250115 02:52:21.236518 4332 raft_consensus.cc:3054] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:21.236945 4487 raft_consensus.cc:2463] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d93156ddb0f2457fa3ba5b43150885ae in term 1.
I20250115 02:52:21.236359 4537 raft_consensus.cc:695] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 1 LEADER]: Becoming Leader. State: Replica: 4b9c97e21437419383aed90eca8a4ffa, State: Running, Role: LEADER
I20250115 02:52:21.238214 4376 leader_election.cc:304] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a34437bd4f3b47368d99913d56ecef7d, d93156ddb0f2457fa3ba5b43150885ae; no voters:
I20250115 02:52:21.239128 4547 raft_consensus.cc:2798] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:21.239077 4537 consensus_queue.cc:237] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:21.242044 4332 raft_consensus.cc:2463] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d93156ddb0f2457fa3ba5b43150885ae in term 1.
I20250115 02:52:21.243937 4547 raft_consensus.cc:695] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 1 LEADER]: Becoming Leader. State: Replica: d93156ddb0f2457fa3ba5b43150885ae, State: Running, Role: LEADER
I20250115 02:52:21.250528 4062 catalog_manager.cc:5526] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b9c97e21437419383aed90eca8a4ffa (127.2.36.65). New cstate: current_term: 1 leader_uuid: "4b9c97e21437419383aed90eca8a4ffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:21.258281 4547 consensus_queue.cc:237] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } }
I20250115 02:52:21.278451 4062 catalog_manager.cc:5526] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae reported cstate change: term changed from 0 to 1, leader changed from <none> to d93156ddb0f2457fa3ba5b43150885ae (127.2.36.66). New cstate: current_term: 1 leader_uuid: "d93156ddb0f2457fa3ba5b43150885ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:21.494321 4537 consensus_queue.cc:1035] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:21.631621 4540 consensus_queue.cc:1035] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:21.667702 4547 consensus_queue.cc:1035] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:21.736599 4557 consensus_queue.cc:1035] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9c97e21437419383aed90eca8a4ffa" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 45083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:21.779198 4537 consensus_queue.cc:1035] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93156ddb0f2457fa3ba5b43150885ae" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:21.802129 4556 consensus_queue.cc:1035] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a34437bd4f3b47368d99913d56ecef7d" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:22.088361 4476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:52:22.106628 4320 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250115 02:52:22.129607 4400 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:52:22.488082 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:52:22.547464 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:22.548084 2193 tablet_replica.cc:331] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa: stopping tablet replica
I20250115 02:52:22.548722 2193 raft_consensus.cc:2238] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:22.549414 2193 raft_consensus.cc:2267] T 65a92db1130849ad9aa850dc4f5c87d8 P 4b9c97e21437419383aed90eca8a4ffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:22.551560 2193 tablet_replica.cc:331] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa: stopping tablet replica
I20250115 02:52:22.552268 2193 raft_consensus.cc:2238] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:22.553165 2193 raft_consensus.cc:2267] T a40ace077e2d43e8bfc88d20a8bb9920 P 4b9c97e21437419383aed90eca8a4ffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:22.555564 2193 tablet_replica.cc:331] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa: stopping tablet replica
I20250115 02:52:22.556614 2193 raft_consensus.cc:2238] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:22.557111 2193 raft_consensus.cc:2267] T d7a566fcd649414e86cfda163a4f1eb4 P 4b9c97e21437419383aed90eca8a4ffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:22.575882 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:52:22.588714 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:52:22.618187 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:22.618917 2193 tablet_replica.cc:331] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae: stopping tablet replica
I20250115 02:52:22.619670 2193 raft_consensus.cc:2238] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:22.620409 2193 raft_consensus.cc:2267] T d7a566fcd649414e86cfda163a4f1eb4 P d93156ddb0f2457fa3ba5b43150885ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:22.622517 2193 tablet_replica.cc:331] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae: stopping tablet replica
I20250115 02:52:22.623066 2193 raft_consensus.cc:2238] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:22.623446 2193 raft_consensus.cc:2267] T 65a92db1130849ad9aa850dc4f5c87d8 P d93156ddb0f2457fa3ba5b43150885ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:22.625618 2193 tablet_replica.cc:331] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae: stopping tablet replica
I20250115 02:52:22.626125 2193 raft_consensus.cc:2238] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:22.626609 2193 raft_consensus.cc:2267] T a40ace077e2d43e8bfc88d20a8bb9920 P d93156ddb0f2457fa3ba5b43150885ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:22.649377 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:52:22.661242 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:52:22.686522 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:22.687110 2193 tablet_replica.cc:331] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d: stopping tablet replica
I20250115 02:52:22.687641 2193 raft_consensus.cc:2238] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:22.688053 2193 raft_consensus.cc:2267] T a40ace077e2d43e8bfc88d20a8bb9920 P a34437bd4f3b47368d99913d56ecef7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:22.689718 2193 tablet_replica.cc:331] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d: stopping tablet replica
I20250115 02:52:22.690198 2193 raft_consensus.cc:2238] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:22.690575 2193 raft_consensus.cc:2267] T d7a566fcd649414e86cfda163a4f1eb4 P a34437bd4f3b47368d99913d56ecef7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:22.692744 2193 tablet_replica.cc:331] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d: stopping tablet replica
I20250115 02:52:22.693272 2193 raft_consensus.cc:2238] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:22.693657 2193 raft_consensus.cc:2267] T 65a92db1130849ad9aa850dc4f5c87d8 P a34437bd4f3b47368d99913d56ecef7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:22.713507 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:52:22.724685 2193 master.cc:537] Master@127.2.36.126:40401 shutting down...
I20250115 02:52:22.741904 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:22.742708 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:22.743098 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 1f7fb9698c994356986f9b8af5dc9d78: stopping tablet replica
I20250115 02:52:22.762293 2193 master.cc:559] Master@127.2.36.126:40401 shutdown complete.
I20250115 02:52:22.773330 2193 master.cc:537] Master@127.2.36.125:37199 shutting down...
I20250115 02:52:22.787851 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:22.788345 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:22.788647 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2d9691daf7bc4374bbe106202f42cb4a: stopping tablet replica
I20250115 02:52:22.806697 2193 master.cc:559] Master@127.2.36.125:37199 shutdown complete.
I20250115 02:52:22.818805 2193 master.cc:537] Master@127.2.36.124:46587 shutting down...
I20250115 02:52:22.834322 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:22.834846 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:22.835122 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 1538993b13f44c1389830d21b5b27e05: stopping tablet replica
I20250115 02:52:22.852825 2193 master.cc:559] Master@127.2.36.124:46587 shutdown complete.
[ OK ] RemoteKsckTest.TestTableConsistency (3181 ms)
[ RUN ] RemoteKsckTest.TestChecksum
I20250115 02:52:22.878975 2193 test_util.cc:274] Using random seed: -1001427223
I20250115 02:52:22.879886 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:46327,127.2.36.125:38755,127.2.36.124:35779
I20250115 02:52:22.881078 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:22.886600 4606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:22.887398 4607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:22.888029 4609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:22.888613 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:22.889425 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:22.889611 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:22.889748 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909542889732 us; error 0 us; skew 500 ppm
I20250115 02:52:22.890244 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:22.898197 2193 webserver.cc:458] Webserver started at http://127.2.36.126:40151/ using document root <none> and password file <none>
I20250115 02:52:22.898700 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:22.898928 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:22.899169 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:22.900202 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "6a9f11e57e1744768c15954645f889e6"
format_stamp: "Formatted at 2025-01-15 02:52:22 on dist-test-slave-1lv2"
I20250115 02:52:22.904367 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:52:22.907418 4614 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:22.908102 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:52:22.908371 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "6a9f11e57e1744768c15954645f889e6"
format_stamp: "Formatted at 2025-01-15 02:52:22 on dist-test-slave-1lv2"
I20250115 02:52:22.908651 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:22.923924 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:22.924893 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:22.957230 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:46327
I20250115 02:52:22.957315 4665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:46327 every 8 connection(s)
I20250115 02:52:22.960796 4666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:22.960974 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:22.966850 4668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:22.967533 4666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:22.967777 4669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:22.969183 4671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:22.970239 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:22.971241 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:22.971432 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:22.971596 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909542971578 us; error 0 us; skew 500 ppm
I20250115 02:52:22.972105 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:22.974812 2193 webserver.cc:458] Webserver started at http://127.2.36.125:39063/ using document root <none> and password file <none>
I20250115 02:52:22.975253 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:22.975422 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:22.975651 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:22.976629 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "6cb1a8ce3bb1464ca03a9efd42275514"
format_stamp: "Formatted at 2025-01-15 02:52:22 on dist-test-slave-1lv2"
I20250115 02:52:22.979961 4666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:22.981576 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
W20250115 02:52:22.985114 4666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:38755: Network error: Client connection negotiation failed: client connection to 127.2.36.125:38755: connect: Connection refused (error 111)
I20250115 02:52:22.985139 4679 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:22.986140 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250115 02:52:22.986410 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "6cb1a8ce3bb1464ca03a9efd42275514"
format_stamp: "Formatted at 2025-01-15 02:52:22 on dist-test-slave-1lv2"
I20250115 02:52:22.986683 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:23.003544 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:23.004530 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:23.006681 4666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } attempt: 1
W20250115 02:52:23.010828 4666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:38755: Network error: Client connection negotiation failed: client connection to 127.2.36.125:38755: connect: Connection refused (error 111)
I20250115 02:52:23.038825 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:38755
I20250115 02:52:23.038897 4730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:38755 every 8 connection(s)
I20250115 02:52:23.042403 4731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:23.042651 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:23.048797 4733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:23.049494 4731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:23.049300 4666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } attempt: 2
W20250115 02:52:23.049861 4734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:23.050621 4736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:23.052654 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:23.053620 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:23.053819 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:23.054008 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909543053991 us; error 0 us; skew 500 ppm
I20250115 02:52:23.054699 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:23.057466 2193 webserver.cc:458] Webserver started at http://127.2.36.124:34943/ using document root <none> and password file <none>
I20250115 02:52:23.057968 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:23.058195 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:23.058507 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:23.059903 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "f9123baf4fcb469ea277265b0ae8195e"
format_stamp: "Formatted at 2025-01-15 02:52:23 on dist-test-slave-1lv2"
I20250115 02:52:23.063870 4666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:23.064908 4731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:23.065467 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
W20250115 02:52:23.068817 4666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:35779: Network error: Client connection negotiation failed: client connection to 127.2.36.124:35779: connect: Connection refused (error 111)
I20250115 02:52:23.070232 4743 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:23.071097 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250115 02:52:23.071440 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "f9123baf4fcb469ea277265b0ae8195e"
format_stamp: "Formatted at 2025-01-15 02:52:23 on dist-test-slave-1lv2"
I20250115 02:52:23.071780 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:23.074848 4731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:23.080260 4731 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:35779: Network error: Client connection negotiation failed: client connection to 127.2.36.124:35779: connect: Connection refused (error 111)
I20250115 02:52:23.092077 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:23.093149 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:23.098304 4666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } attempt: 1
W20250115 02:52:23.102841 4666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:35779: Network error: Client connection negotiation failed: client connection to 127.2.36.124:35779: connect: Connection refused (error 111)
I20250115 02:52:23.128188 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:35779
I20250115 02:52:23.128281 4795 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:35779 every 8 connection(s)
I20250115 02:52:23.131716 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:52:23.132654 4796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:23.138290 4796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:23.143682 4731 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } attempt: 1
I20250115 02:52:23.148581 4796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:23.157784 4796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:23.160387 4731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514: Bootstrap starting.
I20250115 02:52:23.165364 4666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } attempt: 2
I20250115 02:52:23.165680 4731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:23.170405 4731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514: No bootstrap required, opened a new log
I20250115 02:52:23.173594 4731 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } }
I20250115 02:52:23.174332 4731 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:23.174626 4731 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb1a8ce3bb1464ca03a9efd42275514, State: Initialized, Role: FOLLOWER
I20250115 02:52:23.175261 4796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e: Bootstrap starting.
I20250115 02:52:23.175449 4731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } }
I20250115 02:52:23.177748 4801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } } }
I20250115 02:52:23.178633 4801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:23.179332 4731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:23.180986 4666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6: Bootstrap starting.
I20250115 02:52:23.181417 4796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:23.186435 4666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:23.187354 4796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e: No bootstrap required, opened a new log
I20250115 02:52:23.189738 4796 raft_consensus.cc:357] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } }
I20250115 02:52:23.190342 4796 raft_consensus.cc:383] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:23.190564 4796 raft_consensus.cc:738] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9123baf4fcb469ea277265b0ae8195e, State: Initialized, Role: FOLLOWER
I20250115 02:52:23.191176 4796 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } }
I20250115 02:52:23.191478 4666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6: No bootstrap required, opened a new log
I20250115 02:52:23.192939 4810 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } } }
I20250115 02:52:23.193998 4810 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:23.194620 4796 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:23.194422 4666 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } }
I20250115 02:52:23.195269 4666 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:23.195570 4666 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9f11e57e1744768c15954645f889e6, State: Initialized, Role: FOLLOWER
I20250115 02:52:23.196275 4666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } }
I20250115 02:52:23.199118 4813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } } }
I20250115 02:52:23.200032 4813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:23.202056 4666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:23.203851 4801 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:23.204371 4801 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } }
I20250115 02:52:23.206568 4801 leader_election.cc:290] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a9f11e57e1744768c15954645f889e6 (127.2.36.126:46327), f9123baf4fcb469ea277265b0ae8195e (127.2.36.124:35779)
I20250115 02:52:23.208432 4641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9f11e57e1744768c15954645f889e6" is_pre_election: true
I20250115 02:52:23.209208 4641 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cb1a8ce3bb1464ca03a9efd42275514 in term 0.
I20250115 02:52:23.210294 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9123baf4fcb469ea277265b0ae8195e" is_pre_election: true
I20250115 02:52:23.211053 4771 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cb1a8ce3bb1464ca03a9efd42275514 in term 0.
I20250115 02:52:23.212831 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a9f11e57e1744768c15954645f889e6, 6cb1a8ce3bb1464ca03a9efd42275514; no voters:
I20250115 02:52:23.214286 4801 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:23.214639 4801 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:23.214990 4801 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:23.223027 4801 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } }
I20250115 02:52:23.226507 4641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9f11e57e1744768c15954645f889e6"
I20250115 02:52:23.227245 4641 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:23.234959 4641 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cb1a8ce3bb1464ca03a9efd42275514 in term 1.
I20250115 02:52:23.237154 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9123baf4fcb469ea277265b0ae8195e"
I20250115 02:52:23.237905 4771 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:23.245488 4771 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cb1a8ce3bb1464ca03a9efd42275514 in term 1.
I20250115 02:52:23.247555 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cb1a8ce3bb1464ca03a9efd42275514, f9123baf4fcb469ea277265b0ae8195e; no voters:
I20250115 02:52:23.252784 4801 leader_election.cc:290] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [CANDIDATE]: Term 1 election: Requested vote from peers 6a9f11e57e1744768c15954645f889e6 (127.2.36.126:46327), f9123baf4fcb469ea277265b0ae8195e (127.2.36.124:35779)
I20250115 02:52:23.254175 4801 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:23.255338 4801 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 1 LEADER]: Becoming Leader. State: Replica: 6cb1a8ce3bb1464ca03a9efd42275514, State: Running, Role: LEADER
I20250115 02:52:23.256242 4801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } }
W20250115 02:52:23.258239 4833 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:23.258549 4833 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:23.265604 4834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb1a8ce3bb1464ca03a9efd42275514. Latest consensus state: current_term: 1 leader_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } } }
I20250115 02:52:23.266651 4834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:23.268564 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:52:23.268865 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250115 02:52:23.270222 4839 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:23.270556 4839 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:23.270640 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:23.270988 4841 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
W20250115 02:52:23.271585 4840 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:23.271857 4840 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:23.277969 4841 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250115 02:52:23.279062 4842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:23.280223 4843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:23.282747 4845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:23.285849 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:23.286917 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:23.287185 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:23.287389 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909543287371 us; error 0 us; skew 500 ppm
I20250115 02:52:23.287333 4771 raft_consensus.cc:1270] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [term 1 FOLLOWER]: Refusing update from remote peer 6cb1a8ce3bb1464ca03a9efd42275514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:23.287482 4641 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [term 1 FOLLOWER]: Refusing update from remote peer 6cb1a8ce3bb1464ca03a9efd42275514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:23.288506 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:23.289129 4801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:23.289875 4834 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:23.292294 2193 webserver.cc:458] Webserver started at http://127.2.36.65:40709/ using document root <none> and password file <none>
I20250115 02:52:23.293218 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:23.293463 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:23.293794 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:23.295594 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "292243cf07b047babbaad8c021b20a3d"
format_stamp: "Formatted at 2025-01-15 02:52:23 on dist-test-slave-1lv2"
I20250115 02:52:23.302666 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250115 02:52:23.304700 4810 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb1a8ce3bb1464ca03a9efd42275514. Latest consensus state: current_term: 1 leader_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } } }
I20250115 02:52:23.305555 4810 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:23.309383 4854 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:23.310601 2193 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250115 02:52:23.311064 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "292243cf07b047babbaad8c021b20a3d"
format_stamp: "Formatted at 2025-01-15 02:52:23 on dist-test-slave-1lv2"
I20250115 02:52:23.311491 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:23.316562 4801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } } }
I20250115 02:52:23.317363 4801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:23.318642 4841 catalog_manager.cc:1348] Generated new cluster ID: 345cd2014d2c4330a62784174194bfe0
I20250115 02:52:23.319044 4841 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:52:23.319555 4810 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } } }
I20250115 02:52:23.320072 4810 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:23.322407 4813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb1a8ce3bb1464ca03a9efd42275514. Latest consensus state: current_term: 1 leader_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } } }
I20250115 02:52:23.323238 4813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:23.331709 4834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } } }
I20250115 02:52:23.332445 4834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:23.337605 4813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f11e57e1744768c15954645f889e6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46327 } } peers { permanent_uuid: "6cb1a8ce3bb1464ca03a9efd42275514" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38755 } } peers { permanent_uuid: "f9123baf4fcb469ea277265b0ae8195e" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 35779 } } }
I20250115 02:52:23.338492 4813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:23.353348 4841 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:52:23.355038 4841 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:52:23.356220 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:23.357601 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:23.359330 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:23.362191 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:23.362433 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:23.362707 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:23.362919 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250115 02:52:23.376396 4841 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514: Generated new TSK 0
I20250115 02:52:23.377947 4841 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:52:23.409947 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:43675
I20250115 02:52:23.410027 4921 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:43675 every 8 connection(s)
I20250115 02:52:23.419845 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:23.428208 4922 heartbeater.cc:346] Connected to a master server at 127.2.36.124:35779
I20250115 02:52:23.428606 4922 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:23.429440 4922 heartbeater.cc:509] Master 127.2.36.124:35779 requested a full tablet report, sending...
I20250115 02:52:23.432096 4923 heartbeater.cc:346] Connected to a master server at 127.2.36.125:38755
I20250115 02:52:23.432428 4761 ts_manager.cc:194] Registered new tserver with Master: 292243cf07b047babbaad8c021b20a3d (127.2.36.65:43675)
W20250115 02:52:23.433326 4929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:23.432646 4923 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:23.434868 4923 heartbeater.cc:509] Master 127.2.36.125:38755 requested a full tablet report, sending...
I20250115 02:52:23.437613 4696 ts_manager.cc:194] Registered new tserver with Master: 292243cf07b047babbaad8c021b20a3d (127.2.36.65:43675)
I20250115 02:52:23.439105 4924 heartbeater.cc:346] Connected to a master server at 127.2.36.126:46327
I20250115 02:52:23.439559 4924 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:23.440419 4924 heartbeater.cc:509] Master 127.2.36.126:46327 requested a full tablet report, sending...
I20250115 02:52:23.440310 4696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35820
I20250115 02:52:23.442536 2193 server_base.cc:1034] running on GCE node
W20250115 02:52:23.441187 4930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:23.443588 4631 ts_manager.cc:194] Registered new tserver with Master: 292243cf07b047babbaad8c021b20a3d (127.2.36.65:43675)
W20250115 02:52:23.443836 4932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:23.445053 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:23.445269 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:23.445430 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909543445417 us; error 0 us; skew 500 ppm
I20250115 02:52:23.446058 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:23.448626 2193 webserver.cc:458] Webserver started at http://127.2.36.66:39841/ using document root <none> and password file <none>
I20250115 02:52:23.449048 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:23.449203 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:23.449400 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:23.450367 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "fb21509e715a4053b9e4c762ef4992c2"
format_stamp: "Formatted at 2025-01-15 02:52:23 on dist-test-slave-1lv2"
I20250115 02:52:23.454548 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250115 02:52:23.455224 4923 heartbeater.cc:501] Master 127.2.36.125:38755 was elected leader, sending a full tablet report...
I20250115 02:52:23.459025 4937 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:23.459864 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250115 02:52:23.460263 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "fb21509e715a4053b9e4c762ef4992c2"
format_stamp: "Formatted at 2025-01-15 02:52:23 on dist-test-slave-1lv2"
I20250115 02:52:23.460536 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:23.474673 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:23.475762 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:23.477075 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:23.479370 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:23.479571 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:23.479799 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:23.479940 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:23.528012 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:38991
I20250115 02:52:23.528111 4999 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:38991 every 8 connection(s)
I20250115 02:52:23.534900 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:23.551678 5008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:23.552590 5001 heartbeater.cc:346] Connected to a master server at 127.2.36.125:38755
I20250115 02:52:23.553057 5001 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:23.553959 5001 heartbeater.cc:509] Master 127.2.36.125:38755 requested a full tablet report, sending...
I20250115 02:52:23.556442 4696 ts_manager.cc:194] Registered new tserver with Master: fb21509e715a4053b9e4c762ef4992c2 (127.2.36.66:38991)
I20250115 02:52:23.558404 4696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35828
I20250115 02:52:23.561141 2193 server_base.cc:1034] running on GCE node
W20250115 02:52:23.561882 5006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:23.562626 5010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:23.563375 5000 heartbeater.cc:346] Connected to a master server at 127.2.36.124:35779
I20250115 02:52:23.563767 5000 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:23.563808 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:23.564198 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:23.564378 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909543564360 us; error 0 us; skew 500 ppm
I20250115 02:52:23.564424 5000 heartbeater.cc:509] Master 127.2.36.124:35779 requested a full tablet report, sending...
I20250115 02:52:23.565156 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:23.566902 5002 heartbeater.cc:346] Connected to a master server at 127.2.36.126:46327
I20250115 02:52:23.567289 5002 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:23.567950 5002 heartbeater.cc:509] Master 127.2.36.126:46327 requested a full tablet report, sending...
I20250115 02:52:23.567297 4761 ts_manager.cc:194] Registered new tserver with Master: fb21509e715a4053b9e4c762ef4992c2 (127.2.36.66:38991)
I20250115 02:52:23.570535 4631 ts_manager.cc:194] Registered new tserver with Master: fb21509e715a4053b9e4c762ef4992c2 (127.2.36.66:38991)
I20250115 02:52:23.572386 5001 heartbeater.cc:501] Master 127.2.36.125:38755 was elected leader, sending a full tablet report...
I20250115 02:52:23.572753 2193 webserver.cc:458] Webserver started at http://127.2.36.67:41263/ using document root <none> and password file <none>
I20250115 02:52:23.573344 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:23.573604 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:23.573858 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:23.574923 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "4a115961c291444a88cdfe123e7b6769"
format_stamp: "Formatted at 2025-01-15 02:52:23 on dist-test-slave-1lv2"
I20250115 02:52:23.579434 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250115 02:52:23.583599 5015 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:23.584455 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250115 02:52:23.584769 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "4a115961c291444a88cdfe123e7b6769"
format_stamp: "Formatted at 2025-01-15 02:52:23 on dist-test-slave-1lv2"
I20250115 02:52:23.585039 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksum.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:23.629976 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:23.631491 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:23.632819 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:23.635123 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:23.635325 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:23.635565 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:23.635720 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:23.696949 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:33531
I20250115 02:52:23.697008 5077 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:33531 every 8 connection(s)
I20250115 02:52:23.723556 5079 heartbeater.cc:346] Connected to a master server at 127.2.36.125:38755
I20250115 02:52:23.724072 5079 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:23.724988 5079 heartbeater.cc:509] Master 127.2.36.125:38755 requested a full tablet report, sending...
I20250115 02:52:23.727223 4695 ts_manager.cc:194] Registered new tserver with Master: 4a115961c291444a88cdfe123e7b6769 (127.2.36.67:33531)
I20250115 02:52:23.728547 4695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35836
I20250115 02:52:23.731180 5078 heartbeater.cc:346] Connected to a master server at 127.2.36.124:35779
I20250115 02:52:23.731487 5078 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:23.732079 5078 heartbeater.cc:509] Master 127.2.36.124:35779 requested a full tablet report, sending...
I20250115 02:52:23.740017 5081 heartbeater.cc:346] Connected to a master server at 127.2.36.126:46327
I20250115 02:52:23.739970 4761 ts_manager.cc:194] Registered new tserver with Master: 4a115961c291444a88cdfe123e7b6769 (127.2.36.67:33531)
I20250115 02:52:23.740773 5081 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:23.741307 5079 heartbeater.cc:501] Master 127.2.36.125:38755 was elected leader, sending a full tablet report...
I20250115 02:52:23.741467 5081 heartbeater.cc:509] Master 127.2.36.126:46327 requested a full tablet report, sending...
I20250115 02:52:23.743661 4631 ts_manager.cc:194] Registered new tserver with Master: 4a115961c291444a88cdfe123e7b6769 (127.2.36.67:33531)
I20250115 02:52:23.743908 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005238417s
I20250115 02:52:23.786289 4694 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35838:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:52:23.788708 4694 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:52:23.914917 5043 tablet_service.cc:1467] Processing CreateTablet for tablet 761e65d9c1a3460dbf5a652b21f0070e (DEFAULT_TABLE table=ksck-test-table [id=d860eeed952444d198200b409e556f21]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:23.916282 5043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 761e65d9c1a3460dbf5a652b21f0070e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:23.914916 5042 tablet_service.cc:1467] Processing CreateTablet for tablet 6b751dbd64a34dfc9a5049db6b52d3c5 (DEFAULT_TABLE table=ksck-test-table [id=d860eeed952444d198200b409e556f21]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:23.921746 4887 tablet_service.cc:1467] Processing CreateTablet for tablet 761e65d9c1a3460dbf5a652b21f0070e (DEFAULT_TABLE table=ksck-test-table [id=d860eeed952444d198200b409e556f21]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:23.922333 5042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b751dbd64a34dfc9a5049db6b52d3c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:23.923069 4887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 761e65d9c1a3460dbf5a652b21f0070e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:23.925488 4886 tablet_service.cc:1467] Processing CreateTablet for tablet 6b751dbd64a34dfc9a5049db6b52d3c5 (DEFAULT_TABLE table=ksck-test-table [id=d860eeed952444d198200b409e556f21]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:23.927147 4886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b751dbd64a34dfc9a5049db6b52d3c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:23.953075 5041 tablet_service.cc:1467] Processing CreateTablet for tablet 8f37e18644d44e0eb6ff57f0101a52ff (DEFAULT_TABLE table=ksck-test-table [id=d860eeed952444d198200b409e556f21]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:23.954397 5041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f37e18644d44e0eb6ff57f0101a52ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:23.964032 5099 tablet_bootstrap.cc:492] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d: Bootstrap starting.
I20250115 02:52:23.966620 4886 tablet_service.cc:1467] Processing CreateTablet for tablet 8f37e18644d44e0eb6ff57f0101a52ff (DEFAULT_TABLE table=ksck-test-table [id=d860eeed952444d198200b409e556f21]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:23.968099 4886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f37e18644d44e0eb6ff57f0101a52ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:23.978626 5099 tablet_bootstrap.cc:654] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:23.987314 4965 tablet_service.cc:1467] Processing CreateTablet for tablet 761e65d9c1a3460dbf5a652b21f0070e (DEFAULT_TABLE table=ksck-test-table [id=d860eeed952444d198200b409e556f21]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:23.987988 4964 tablet_service.cc:1467] Processing CreateTablet for tablet 6b751dbd64a34dfc9a5049db6b52d3c5 (DEFAULT_TABLE table=ksck-test-table [id=d860eeed952444d198200b409e556f21]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:23.988261 4963 tablet_service.cc:1467] Processing CreateTablet for tablet 8f37e18644d44e0eb6ff57f0101a52ff (DEFAULT_TABLE table=ksck-test-table [id=d860eeed952444d198200b409e556f21]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:23.990243 4963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f37e18644d44e0eb6ff57f0101a52ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:23.994264 5100 tablet_bootstrap.cc:492] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769: Bootstrap starting.
I20250115 02:52:23.999158 4964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b751dbd64a34dfc9a5049db6b52d3c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:24.005265 4965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 761e65d9c1a3460dbf5a652b21f0070e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:24.008893 5100 tablet_bootstrap.cc:654] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:24.045557 5100 tablet_bootstrap.cc:492] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769: No bootstrap required, opened a new log
I20250115 02:52:24.046180 5100 ts_tablet_manager.cc:1397] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.000s
I20250115 02:52:24.052412 5100 raft_consensus.cc:357] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } } peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } }
I20250115 02:52:24.053386 5100 raft_consensus.cc:383] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:24.059727 5100 raft_consensus.cc:738] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a115961c291444a88cdfe123e7b6769, State: Initialized, Role: FOLLOWER
I20250115 02:52:24.060489 5100 consensus_queue.cc:260] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } } peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } }
I20250115 02:52:24.063330 5104 tablet_bootstrap.cc:492] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2: Bootstrap starting.
I20250115 02:52:24.064029 5100 ts_tablet_manager.cc:1428] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250115 02:52:24.065616 5099 tablet_bootstrap.cc:492] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d: No bootstrap required, opened a new log
I20250115 02:52:24.073298 5099 ts_tablet_manager.cc:1397] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d: Time spent bootstrapping tablet: real 0.109s user 0.036s sys 0.018s
I20250115 02:52:24.076377 5099 raft_consensus.cc:357] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.077220 5099 raft_consensus.cc:383] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:24.074551 5100 tablet_bootstrap.cc:492] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769: Bootstrap starting.
I20250115 02:52:24.077579 5099 raft_consensus.cc:738] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 292243cf07b047babbaad8c021b20a3d, State: Initialized, Role: FOLLOWER
I20250115 02:52:24.078402 5099 consensus_queue.cc:260] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.083521 5100 tablet_bootstrap.cc:654] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:24.094535 5100 tablet_bootstrap.cc:492] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769: No bootstrap required, opened a new log
I20250115 02:52:24.095184 5100 ts_tablet_manager.cc:1397] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.009s
I20250115 02:52:24.096462 5099 ts_tablet_manager.cc:1428] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d: Time spent starting tablet: real 0.023s user 0.016s sys 0.003s
I20250115 02:52:24.098098 5100 raft_consensus.cc:357] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.100577 5100 raft_consensus.cc:383] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:24.100947 5100 raft_consensus.cc:738] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a115961c291444a88cdfe123e7b6769, State: Initialized, Role: FOLLOWER
I20250115 02:52:24.101539 5100 consensus_queue.cc:260] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.105060 5099 tablet_bootstrap.cc:492] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d: Bootstrap starting.
I20250115 02:52:24.105778 5100 ts_tablet_manager.cc:1428] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250115 02:52:24.109596 5100 tablet_bootstrap.cc:492] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769: Bootstrap starting.
I20250115 02:52:24.111590 5099 tablet_bootstrap.cc:654] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:24.115770 5100 tablet_bootstrap.cc:654] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:24.128111 5099 tablet_bootstrap.cc:492] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d: No bootstrap required, opened a new log
I20250115 02:52:24.128738 5099 ts_tablet_manager.cc:1397] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.005s
I20250115 02:52:24.128880 5100 tablet_bootstrap.cc:492] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769: No bootstrap required, opened a new log
I20250115 02:52:24.129500 5100 ts_tablet_manager.cc:1397] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.009s
I20250115 02:52:24.131609 5099 raft_consensus.cc:357] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } } peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } }
I20250115 02:52:24.132583 5099 raft_consensus.cc:383] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:24.132233 5100 raft_consensus.cc:357] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.132993 5099 raft_consensus.cc:738] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 292243cf07b047babbaad8c021b20a3d, State: Initialized, Role: FOLLOWER
I20250115 02:52:24.133193 5100 raft_consensus.cc:383] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:24.133872 5099 consensus_queue.cc:260] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } } peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } }
I20250115 02:52:24.133646 5100 raft_consensus.cc:738] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a115961c291444a88cdfe123e7b6769, State: Initialized, Role: FOLLOWER
I20250115 02:52:24.136754 5100 consensus_queue.cc:260] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.137749 5099 ts_tablet_manager.cc:1428] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250115 02:52:24.142611 5099 tablet_bootstrap.cc:492] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d: Bootstrap starting.
I20250115 02:52:24.139499 5100 ts_tablet_manager.cc:1428] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250115 02:52:24.144902 5105 raft_consensus.cc:491] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:24.145414 5105 raft_consensus.cc:513] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.148635 5099 tablet_bootstrap.cc:654] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:24.148916 5105 leader_election.cc:290] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb21509e715a4053b9e4c762ef4992c2 (127.2.36.66:38991), 4a115961c291444a88cdfe123e7b6769 (127.2.36.67:33531)
I20250115 02:52:24.099130 5104 tablet_bootstrap.cc:654] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:24.181783 5104 tablet_bootstrap.cc:492] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2: No bootstrap required, opened a new log
I20250115 02:52:24.182359 5104 ts_tablet_manager.cc:1397] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2: Time spent bootstrapping tablet: real 0.119s user 0.015s sys 0.003s
I20250115 02:52:24.184058 5099 tablet_bootstrap.cc:492] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d: No bootstrap required, opened a new log
I20250115 02:52:24.188110 5099 ts_tablet_manager.cc:1397] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.009s
I20250115 02:52:24.186020 4975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b751dbd64a34dfc9a5049db6b52d3c5" candidate_uuid: "292243cf07b047babbaad8c021b20a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb21509e715a4053b9e4c762ef4992c2" is_pre_election: true
W20250115 02:52:24.190953 4857 leader_election.cc:343] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fb21509e715a4053b9e4c762ef4992c2 (127.2.36.66:38991): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:52:24.192134 5099 raft_consensus.cc:357] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.192945 5099 raft_consensus.cc:383] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:24.193275 5099 raft_consensus.cc:738] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 292243cf07b047babbaad8c021b20a3d, State: Initialized, Role: FOLLOWER
I20250115 02:52:24.194072 5099 consensus_queue.cc:260] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.195147 5053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b751dbd64a34dfc9a5049db6b52d3c5" candidate_uuid: "292243cf07b047babbaad8c021b20a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a115961c291444a88cdfe123e7b6769" is_pre_election: true
I20250115 02:52:24.196002 5053 raft_consensus.cc:2463] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 292243cf07b047babbaad8c021b20a3d in term 0.
I20250115 02:52:24.196609 5099 ts_tablet_manager.cc:1428] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250115 02:52:24.197116 4860 leader_election.cc:304] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 292243cf07b047babbaad8c021b20a3d, 4a115961c291444a88cdfe123e7b6769; no voters: fb21509e715a4053b9e4c762ef4992c2
I20250115 02:52:24.199347 5105 raft_consensus.cc:2798] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:24.199802 5105 raft_consensus.cc:491] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:24.200188 5105 raft_consensus.cc:3054] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:24.205602 5104 raft_consensus.cc:357] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.209192 5104 raft_consensus.cc:383] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:24.210505 5104 raft_consensus.cc:738] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb21509e715a4053b9e4c762ef4992c2, State: Initialized, Role: FOLLOWER
I20250115 02:52:24.211338 5104 consensus_queue.cc:260] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.208825 5105 raft_consensus.cc:513] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.213297 5105 leader_election.cc:290] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [CANDIDATE]: Term 1 election: Requested vote from peers fb21509e715a4053b9e4c762ef4992c2 (127.2.36.66:38991), 4a115961c291444a88cdfe123e7b6769 (127.2.36.67:33531)
I20250115 02:52:24.214090 4975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b751dbd64a34dfc9a5049db6b52d3c5" candidate_uuid: "292243cf07b047babbaad8c021b20a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb21509e715a4053b9e4c762ef4992c2"
I20250115 02:52:24.214646 5053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b751dbd64a34dfc9a5049db6b52d3c5" candidate_uuid: "292243cf07b047babbaad8c021b20a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a115961c291444a88cdfe123e7b6769"
I20250115 02:52:24.215942 5053 raft_consensus.cc:3054] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Advancing to term 1
W20250115 02:52:24.217801 4857 leader_election.cc:343] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fb21509e715a4053b9e4c762ef4992c2 (127.2.36.66:38991): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:52:24.222740 5053 raft_consensus.cc:2463] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 292243cf07b047babbaad8c021b20a3d in term 1.
I20250115 02:52:24.224361 4860 leader_election.cc:304] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 292243cf07b047babbaad8c021b20a3d, 4a115961c291444a88cdfe123e7b6769; no voters: fb21509e715a4053b9e4c762ef4992c2
I20250115 02:52:24.228817 5105 raft_consensus.cc:2798] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:24.229709 5105 raft_consensus.cc:695] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 1 LEADER]: Becoming Leader. State: Replica: 292243cf07b047babbaad8c021b20a3d, State: Running, Role: LEADER
I20250115 02:52:24.230661 5105 consensus_queue.cc:237] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.238216 5104 ts_tablet_manager.cc:1428] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2: Time spent starting tablet: real 0.055s user 0.015s sys 0.010s
I20250115 02:52:24.239470 5104 tablet_bootstrap.cc:492] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2: Bootstrap starting.
I20250115 02:52:24.251667 5104 tablet_bootstrap.cc:654] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:24.258224 4696 catalog_manager.cc:5526] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 292243cf07b047babbaad8c021b20a3d (127.2.36.65). New cstate: current_term: 1 leader_uuid: "292243cf07b047babbaad8c021b20a3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:24.263279 5104 tablet_bootstrap.cc:492] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2: No bootstrap required, opened a new log
I20250115 02:52:24.263824 5104 ts_tablet_manager.cc:1397] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.003s
I20250115 02:52:24.264184 4833 catalog_manager.cc:1260] Loaded cluster ID: 345cd2014d2c4330a62784174194bfe0
I20250115 02:52:24.264487 4833 catalog_manager.cc:1553] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e: loading cluster ID for follower catalog manager: success
I20250115 02:52:24.267853 5104 raft_consensus.cc:357] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } } peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } }
I20250115 02:52:24.270203 5104 raft_consensus.cc:383] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:24.270661 5104 raft_consensus.cc:738] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb21509e715a4053b9e4c762ef4992c2, State: Initialized, Role: FOLLOWER
I20250115 02:52:24.271449 5104 consensus_queue.cc:260] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } } peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } }
I20250115 02:52:24.274624 5104 ts_tablet_manager.cc:1428] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250115 02:52:24.275781 5104 tablet_bootstrap.cc:492] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2: Bootstrap starting.
I20250115 02:52:24.281533 5104 tablet_bootstrap.cc:654] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:24.277714 4840 catalog_manager.cc:1260] Loaded cluster ID: 345cd2014d2c4330a62784174194bfe0
I20250115 02:52:24.282166 4840 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6: loading cluster ID for follower catalog manager: success
I20250115 02:52:24.299418 4833 catalog_manager.cc:1575] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e: acquiring CA information for follower catalog manager: success
I20250115 02:52:24.304391 4840 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6: acquiring CA information for follower catalog manager: success
I20250115 02:52:24.305449 4833 catalog_manager.cc:1603] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:24.311272 5104 tablet_bootstrap.cc:492] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2: No bootstrap required, opened a new log
I20250115 02:52:24.311929 5104 ts_tablet_manager.cc:1397] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2: Time spent bootstrapping tablet: real 0.036s user 0.017s sys 0.014s
I20250115 02:52:24.314508 4840 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:24.314710 5104 raft_consensus.cc:357] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.316505 5104 raft_consensus.cc:383] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:24.316903 5104 raft_consensus.cc:738] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb21509e715a4053b9e4c762ef4992c2, State: Initialized, Role: FOLLOWER
I20250115 02:52:24.317641 5104 consensus_queue.cc:260] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.320108 5104 ts_tablet_manager.cc:1428] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250115 02:52:24.320852 5105 raft_consensus.cc:491] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:24.321350 5105 raft_consensus.cc:513] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.323324 5105 leader_election.cc:290] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb21509e715a4053b9e4c762ef4992c2 (127.2.36.66:38991), 4a115961c291444a88cdfe123e7b6769 (127.2.36.67:33531)
I20250115 02:52:24.326107 5053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f37e18644d44e0eb6ff57f0101a52ff" candidate_uuid: "292243cf07b047babbaad8c021b20a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a115961c291444a88cdfe123e7b6769" is_pre_election: true
I20250115 02:52:24.326818 5053 raft_consensus.cc:2463] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 292243cf07b047babbaad8c021b20a3d in term 0.
I20250115 02:52:24.328032 4860 leader_election.cc:304] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 292243cf07b047babbaad8c021b20a3d, 4a115961c291444a88cdfe123e7b6769; no voters:
I20250115 02:52:24.328796 4975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f37e18644d44e0eb6ff57f0101a52ff" candidate_uuid: "292243cf07b047babbaad8c021b20a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb21509e715a4053b9e4c762ef4992c2" is_pre_election: true
I20250115 02:52:24.329111 5105 raft_consensus.cc:2798] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:24.329607 5105 raft_consensus.cc:491] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:24.330029 5105 raft_consensus.cc:3054] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:24.329649 4975 raft_consensus.cc:2463] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 292243cf07b047babbaad8c021b20a3d in term 0.
I20250115 02:52:24.338017 5105 raft_consensus.cc:513] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.340634 4975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f37e18644d44e0eb6ff57f0101a52ff" candidate_uuid: "292243cf07b047babbaad8c021b20a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb21509e715a4053b9e4c762ef4992c2"
I20250115 02:52:24.341149 4975 raft_consensus.cc:3054] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:24.340691 5105 leader_election.cc:290] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [CANDIDATE]: Term 1 election: Requested vote from peers fb21509e715a4053b9e4c762ef4992c2 (127.2.36.66:38991), 4a115961c291444a88cdfe123e7b6769 (127.2.36.67:33531)
I20250115 02:52:24.343271 5053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f37e18644d44e0eb6ff57f0101a52ff" candidate_uuid: "292243cf07b047babbaad8c021b20a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a115961c291444a88cdfe123e7b6769"
I20250115 02:52:24.343793 5053 raft_consensus.cc:3054] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:24.350083 4975 raft_consensus.cc:2463] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 292243cf07b047babbaad8c021b20a3d in term 1.
I20250115 02:52:24.350654 5053 raft_consensus.cc:2463] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 292243cf07b047babbaad8c021b20a3d in term 1.
I20250115 02:52:24.351353 4857 leader_election.cc:304] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 292243cf07b047babbaad8c021b20a3d, fb21509e715a4053b9e4c762ef4992c2; no voters:
I20250115 02:52:24.352212 5105 raft_consensus.cc:2798] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:24.353150 5105 raft_consensus.cc:695] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 1 LEADER]: Becoming Leader. State: Replica: 292243cf07b047babbaad8c021b20a3d, State: Running, Role: LEADER
I20250115 02:52:24.354370 5105 consensus_queue.cc:237] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.368458 4695 catalog_manager.cc:5526] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 292243cf07b047babbaad8c021b20a3d (127.2.36.65). New cstate: current_term: 1 leader_uuid: "292243cf07b047babbaad8c021b20a3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:24.408951 5114 raft_consensus.cc:491] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:24.409355 5114 raft_consensus.cc:513] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } } peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } }
I20250115 02:52:24.411695 5114 leader_election.cc:290] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 292243cf07b047babbaad8c021b20a3d (127.2.36.65:43675), 4a115961c291444a88cdfe123e7b6769 (127.2.36.67:33531)
I20250115 02:52:24.429935 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b751dbd64a34dfc9a5049db6b52d3c5" candidate_uuid: "fb21509e715a4053b9e4c762ef4992c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "292243cf07b047babbaad8c021b20a3d" is_pre_election: true
I20250115 02:52:24.435215 5053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b751dbd64a34dfc9a5049db6b52d3c5" candidate_uuid: "fb21509e715a4053b9e4c762ef4992c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a115961c291444a88cdfe123e7b6769" is_pre_election: true
I20250115 02:52:24.436053 5053 raft_consensus.cc:2388] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fb21509e715a4053b9e4c762ef4992c2 in current term 1: Already voted for candidate 292243cf07b047babbaad8c021b20a3d in this term.
I20250115 02:52:24.437155 4940 leader_election.cc:304] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb21509e715a4053b9e4c762ef4992c2; no voters: 292243cf07b047babbaad8c021b20a3d, 4a115961c291444a88cdfe123e7b6769
I20250115 02:52:24.438304 5114 raft_consensus.cc:3054] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:24.444998 5114 raft_consensus.cc:2743] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:52:24.484722 5103 raft_consensus.cc:491] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:24.485255 5103 raft_consensus.cc:513] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } } peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } }
I20250115 02:52:24.487514 5103 leader_election.cc:290] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 292243cf07b047babbaad8c021b20a3d (127.2.36.65:43675), fb21509e715a4053b9e4c762ef4992c2 (127.2.36.66:38991)
I20250115 02:52:24.499764 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e65d9c1a3460dbf5a652b21f0070e" candidate_uuid: "4a115961c291444a88cdfe123e7b6769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "292243cf07b047babbaad8c021b20a3d" is_pre_election: true
I20250115 02:52:24.500466 4897 raft_consensus.cc:2463] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a115961c291444a88cdfe123e7b6769 in term 0.
I20250115 02:52:24.501694 5016 leader_election.cc:304] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 292243cf07b047babbaad8c021b20a3d, 4a115961c291444a88cdfe123e7b6769; no voters:
I20250115 02:52:24.502518 5103 raft_consensus.cc:2798] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:24.502921 5103 raft_consensus.cc:491] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:24.503249 5103 raft_consensus.cc:3054] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:24.504258 4975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e65d9c1a3460dbf5a652b21f0070e" candidate_uuid: "4a115961c291444a88cdfe123e7b6769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb21509e715a4053b9e4c762ef4992c2" is_pre_election: true
I20250115 02:52:24.504905 4975 raft_consensus.cc:2463] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a115961c291444a88cdfe123e7b6769 in term 0.
I20250115 02:52:24.509169 5103 raft_consensus.cc:513] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } } peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } }
I20250115 02:52:24.511091 5103 leader_election.cc:290] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [CANDIDATE]: Term 1 election: Requested vote from peers 292243cf07b047babbaad8c021b20a3d (127.2.36.65:43675), fb21509e715a4053b9e4c762ef4992c2 (127.2.36.66:38991)
I20250115 02:52:24.512099 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e65d9c1a3460dbf5a652b21f0070e" candidate_uuid: "4a115961c291444a88cdfe123e7b6769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "292243cf07b047babbaad8c021b20a3d"
I20250115 02:52:24.512707 4897 raft_consensus.cc:3054] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:24.515096 4975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e65d9c1a3460dbf5a652b21f0070e" candidate_uuid: "4a115961c291444a88cdfe123e7b6769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb21509e715a4053b9e4c762ef4992c2"
I20250115 02:52:24.515683 4975 raft_consensus.cc:3054] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:24.527228 4975 raft_consensus.cc:2463] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a115961c291444a88cdfe123e7b6769 in term 1.
I20250115 02:52:24.527364 4897 raft_consensus.cc:2463] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a115961c291444a88cdfe123e7b6769 in term 1.
I20250115 02:52:24.528270 5016 leader_election.cc:304] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a115961c291444a88cdfe123e7b6769, fb21509e715a4053b9e4c762ef4992c2; no voters:
I20250115 02:52:24.529018 5103 raft_consensus.cc:2798] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:24.531059 5103 raft_consensus.cc:695] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 1 LEADER]: Becoming Leader. State: Replica: 4a115961c291444a88cdfe123e7b6769, State: Running, Role: LEADER
I20250115 02:52:24.531884 5103 consensus_queue.cc:237] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } } peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } }
I20250115 02:52:24.544049 4695 catalog_manager.cc:5526] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a115961c291444a88cdfe123e7b6769 (127.2.36.67). New cstate: current_term: 1 leader_uuid: "4a115961c291444a88cdfe123e7b6769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:24.678460 5105 consensus_queue.cc:1035] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:24.700073 5105 consensus_queue.cc:1035] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:24.724664 2193 ksck_remote-test.cc:376] Generating row writes...
I20250115 02:52:24.754719 5105 consensus_queue.cc:1035] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:24.771238 5113 consensus_queue.cc:1035] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a115961c291444a88cdfe123e7b6769" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:24.850261 4897 raft_consensus.cc:1270] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d [term 1 FOLLOWER]: Refusing update from remote peer 4a115961c291444a88cdfe123e7b6769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:24.851830 4973 raft_consensus.cc:1270] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2 [term 1 FOLLOWER]: Refusing update from remote peer 4a115961c291444a88cdfe123e7b6769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:24.861157 5125 consensus_queue.cc:1035] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb21509e715a4053b9e4c762ef4992c2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:24.864302 5103 consensus_queue.cc:1035] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "292243cf07b047babbaad8c021b20a3d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:25.596329 5041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:52:25.627568 4963 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:52:25.631004 4886 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250115 02:52:26.034838 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:52:26.172070 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:26.176071 2193 tablet_replica.cc:331] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d: stopping tablet replica
I20250115 02:52:26.177944 2193 raft_consensus.cc:2238] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:26.180538 2193 raft_consensus.cc:2267] T 8f37e18644d44e0eb6ff57f0101a52ff P 292243cf07b047babbaad8c021b20a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:26.186653 2193 tablet_replica.cc:331] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d: stopping tablet replica
I20250115 02:52:26.188288 2193 raft_consensus.cc:2238] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:26.189632 2193 raft_consensus.cc:2267] T 761e65d9c1a3460dbf5a652b21f0070e P 292243cf07b047babbaad8c021b20a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:26.193387 2193 tablet_replica.cc:331] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d: stopping tablet replica
I20250115 02:52:26.194136 2193 raft_consensus.cc:2238] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:26.195685 2193 raft_consensus.cc:2267] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 292243cf07b047babbaad8c021b20a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:26.221788 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:52:26.250767 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:52:26.324874 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:26.325577 2193 tablet_replica.cc:331] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2: stopping tablet replica
I20250115 02:52:26.326300 2193 raft_consensus.cc:2238] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:26.326943 2193 raft_consensus.cc:2267] T 6b751dbd64a34dfc9a5049db6b52d3c5 P fb21509e715a4053b9e4c762ef4992c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:26.329200 2193 tablet_replica.cc:331] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2: stopping tablet replica
I20250115 02:52:26.329823 2193 raft_consensus.cc:2238] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:26.330575 2193 raft_consensus.cc:2267] T 761e65d9c1a3460dbf5a652b21f0070e P fb21509e715a4053b9e4c762ef4992c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:26.333312 2193 tablet_replica.cc:331] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2: stopping tablet replica
I20250115 02:52:26.333936 2193 raft_consensus.cc:2238] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:26.334826 2193 raft_consensus.cc:2267] T 8f37e18644d44e0eb6ff57f0101a52ff P fb21509e715a4053b9e4c762ef4992c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:26.350970 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
W20250115 02:52:26.367467 5016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43675: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20250115 02:52:26.374235 5016 consensus_peers.cc:487] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 -> Peer 292243cf07b047babbaad8c021b20a3d (127.2.36.65:43675): Couldn't send request to peer 292243cf07b047babbaad8c021b20a3d. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43675: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:26.378629 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:52:26.439852 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:26.440703 2193 tablet_replica.cc:331] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769: stopping tablet replica
I20250115 02:52:26.441709 2193 raft_consensus.cc:2238] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:26.442485 2193 raft_consensus.cc:2267] T 8f37e18644d44e0eb6ff57f0101a52ff P 4a115961c291444a88cdfe123e7b6769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:26.445816 2193 tablet_replica.cc:331] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769: stopping tablet replica
I20250115 02:52:26.446470 2193 raft_consensus.cc:2238] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:26.447485 2193 raft_consensus.cc:2267] T 6b751dbd64a34dfc9a5049db6b52d3c5 P 4a115961c291444a88cdfe123e7b6769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:26.450498 2193 tablet_replica.cc:331] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769: stopping tablet replica
I20250115 02:52:26.451256 2193 raft_consensus.cc:2238] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 1 LEADER]: Raft consensus shutting down.
W20250115 02:52:26.451380 5016 consensus_peers.cc:487] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 -> Peer fb21509e715a4053b9e4c762ef4992c2 (127.2.36.66:38991): Couldn't send request to peer fb21509e715a4053b9e4c762ef4992c2. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.66:38991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:26.452549 2193 raft_consensus.cc:2267] T 761e65d9c1a3460dbf5a652b21f0070e P 4a115961c291444a88cdfe123e7b6769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:26.468504 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:52:26.494256 2193 master.cc:537] Master@127.2.36.126:46327 shutting down...
I20250115 02:52:26.524652 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:26.525313 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:26.525641 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6a9f11e57e1744768c15954645f889e6: stopping tablet replica
I20250115 02:52:26.548071 2193 master.cc:559] Master@127.2.36.126:46327 shutdown complete.
I20250115 02:52:26.567739 2193 master.cc:537] Master@127.2.36.125:38755 shutting down...
I20250115 02:52:26.596784 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:26.597698 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:26.598107 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6cb1a8ce3bb1464ca03a9efd42275514: stopping tablet replica
I20250115 02:52:26.620467 2193 master.cc:559] Master@127.2.36.125:38755 shutdown complete.
I20250115 02:52:26.639146 2193 master.cc:537] Master@127.2.36.124:35779 shutting down...
I20250115 02:52:26.667029 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:26.667631 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:26.667948 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P f9123baf4fcb469ea277265b0ae8195e: stopping tablet replica
I20250115 02:52:26.688491 2193 master.cc:559] Master@127.2.36.124:35779 shutdown complete.
[ OK ] RemoteKsckTest.TestChecksum (3861 ms)
[ RUN ] RemoteKsckTest.TestChecksumTimeout
I20250115 02:52:26.742342 2193 test_util.cc:274] Using random seed: -997563858
I20250115 02:52:26.743819 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:46751,127.2.36.125:46189,127.2.36.124:42663
I20250115 02:52:26.745656 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:26.753027 5192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:26.753468 5193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:26.754462 5195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:26.755275 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:26.756484 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:26.756757 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:26.756939 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909546756924 us; error 0 us; skew 500 ppm
I20250115 02:52:26.757611 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:26.760478 2193 webserver.cc:458] Webserver started at http://127.2.36.126:44105/ using document root <none> and password file <none>
I20250115 02:52:26.761099 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:26.761318 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:26.761636 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:26.763247 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "1aacffc182d849b198c02d89fb617bfc"
format_stamp: "Formatted at 2025-01-15 02:52:26 on dist-test-slave-1lv2"
I20250115 02:52:26.769023 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250115 02:52:26.773183 5200 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:26.773985 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:52:26.774314 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "1aacffc182d849b198c02d89fb617bfc"
format_stamp: "Formatted at 2025-01-15 02:52:26 on dist-test-slave-1lv2"
I20250115 02:52:26.774660 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:26.798949 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:26.800480 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:26.847033 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:46751
I20250115 02:52:26.847095 5251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:46751 every 8 connection(s)
I20250115 02:52:26.852027 5252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:26.852191 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:26.859385 5254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:26.860014 5252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:26.860476 5255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:26.870900 5257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:26.879688 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:26.880857 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:26.881134 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:26.881317 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909546881300 us; error 0 us; skew 500 ppm
I20250115 02:52:26.881995 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:26.886489 2193 webserver.cc:458] Webserver started at http://127.2.36.125:36071/ using document root <none> and password file <none>
I20250115 02:52:26.887197 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:26.887393 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:26.887733 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:26.888533 5252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:26.889019 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "b016f34775574b05adbf5dbd2638c133"
format_stamp: "Formatted at 2025-01-15 02:52:26 on dist-test-slave-1lv2"
W20250115 02:52:26.894333 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:46189: Network error: Client connection negotiation failed: client connection to 127.2.36.125:46189: connect: Connection refused (error 111)
I20250115 02:52:26.895861 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20250115 02:52:26.899638 5265 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:26.900547 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250115 02:52:26.900837 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "b016f34775574b05adbf5dbd2638c133"
format_stamp: "Formatted at 2025-01-15 02:52:26 on dist-test-slave-1lv2"
I20250115 02:52:26.901100 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:26.922716 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:26.923893 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:26.948900 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } attempt: 1
W20250115 02:52:26.954133 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:46189: Network error: Client connection negotiation failed: client connection to 127.2.36.125:46189: connect: Connection refused (error 111)
I20250115 02:52:26.967981 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:46189
I20250115 02:52:26.968130 5316 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:46189 every 8 connection(s)
I20250115 02:52:26.972110 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:26.972240 5317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 02:52:26.979239 5319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:26.980466 5320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:26.982635 5317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:26.983111 5322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:26.985015 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:26.986438 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:26.986718 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:26.986977 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909546986956 us; error 0 us; skew 500 ppm
I20250115 02:52:26.987582 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:26.993911 2193 webserver.cc:458] Webserver started at http://127.2.36.124:43499/ using document root <none> and password file <none>
I20250115 02:52:26.994474 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:26.994769 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:26.995117 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:26.996480 5317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:26.996598 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "1b823b6b22994cc59c0ea6509ef4cf91"
format_stamp: "Formatted at 2025-01-15 02:52:26 on dist-test-slave-1lv2"
I20250115 02:52:27.002646 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } attempt: 2
I20250115 02:52:27.003338 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250115 02:52:27.015266 5317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:27.019512 5330 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:27.020403 5252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:27.021899 2193 fs_manager.cc:730] Time spent opening block manager: real 0.016s user 0.005s sys 0.009s
I20250115 02:52:27.022279 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "1b823b6b22994cc59c0ea6509ef4cf91"
format_stamp: "Formatted at 2025-01-15 02:52:26 on dist-test-slave-1lv2"
I20250115 02:52:27.022643 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250115 02:52:27.023162 5317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:42663: Network error: Client connection negotiation failed: client connection to 127.2.36.124:42663: connect: Connection refused (error 111)
W20250115 02:52:27.025116 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:42663: Network error: Client connection negotiation failed: client connection to 127.2.36.124:42663: connect: Connection refused (error 111)
I20250115 02:52:27.048348 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:27.049964 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:27.058588 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } attempt: 1
W20250115 02:52:27.064088 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:42663: Network error: Client connection negotiation failed: client connection to 127.2.36.124:42663: connect: Connection refused (error 111)
I20250115 02:52:27.084672 5317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } attempt: 1
W20250115 02:52:27.090085 5317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:42663: Network error: Client connection negotiation failed: client connection to 127.2.36.124:42663: connect: Connection refused (error 111)
I20250115 02:52:27.102345 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:42663
I20250115 02:52:27.102639 5382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:42663 every 8 connection(s)
I20250115 02:52:27.105623 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:52:27.106937 5383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:27.115705 5383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:27.121626 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } attempt: 2
I20250115 02:52:27.126606 5317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } attempt: 2
I20250115 02:52:27.131366 5383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:27.152606 5383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:27.159948 5317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133: Bootstrap starting.
I20250115 02:52:27.163379 5252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc: Bootstrap starting.
I20250115 02:52:27.167305 5317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:27.170451 5252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:27.176119 5383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91: Bootstrap starting.
I20250115 02:52:27.179729 5252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc: No bootstrap required, opened a new log
I20250115 02:52:27.180308 5317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133: No bootstrap required, opened a new log
I20250115 02:52:27.183202 5252 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } }
I20250115 02:52:27.184064 5252 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:27.183354 5317 raft_consensus.cc:357] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } }
I20250115 02:52:27.184442 5252 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aacffc182d849b198c02d89fb617bfc, State: Initialized, Role: FOLLOWER
I20250115 02:52:27.184540 5317 raft_consensus.cc:383] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:27.184156 5383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:27.184883 5317 raft_consensus.cc:738] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b016f34775574b05adbf5dbd2638c133, State: Initialized, Role: FOLLOWER
I20250115 02:52:27.185354 5252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } }
I20250115 02:52:27.185952 5317 consensus_queue.cc:260] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } }
I20250115 02:52:27.191547 5392 sys_catalog.cc:455] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } } }
I20250115 02:52:27.195376 5392 sys_catalog.cc:458] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:27.194953 5317 sys_catalog.cc:564] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:27.200042 5391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } } }
I20250115 02:52:27.201503 5391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:27.201774 5252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:27.203917 5383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91: No bootstrap required, opened a new log
I20250115 02:52:27.206869 5383 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } }
I20250115 02:52:27.207772 5383 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:27.208114 5383 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b823b6b22994cc59c0ea6509ef4cf91, State: Initialized, Role: FOLLOWER
I20250115 02:52:27.210191 5383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } }
I20250115 02:52:27.215618 5397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } } }
I20250115 02:52:27.219372 5397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:27.218943 5383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:27.250267 5392 raft_consensus.cc:491] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:27.250847 5392 raft_consensus.cc:513] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } }
I20250115 02:52:27.253113 5392 leader_election.cc:290] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1aacffc182d849b198c02d89fb617bfc (127.2.36.126:46751), 1b823b6b22994cc59c0ea6509ef4cf91 (127.2.36.124:42663)
I20250115 02:52:27.256040 5227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b016f34775574b05adbf5dbd2638c133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aacffc182d849b198c02d89fb617bfc" is_pre_election: true
I20250115 02:52:27.256996 5227 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b016f34775574b05adbf5dbd2638c133 in term 0.
I20250115 02:52:27.258250 5358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b016f34775574b05adbf5dbd2638c133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" is_pre_election: true
I20250115 02:52:27.259001 5358 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b016f34775574b05adbf5dbd2638c133 in term 0.
I20250115 02:52:27.260706 5268 leader_election.cc:304] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b823b6b22994cc59c0ea6509ef4cf91, b016f34775574b05adbf5dbd2638c133; no voters:
I20250115 02:52:27.262705 5392 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:27.263222 5392 raft_consensus.cc:491] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:27.263654 5392 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:27.271288 5392 raft_consensus.cc:513] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } }
I20250115 02:52:27.276738 5227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b016f34775574b05adbf5dbd2638c133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aacffc182d849b198c02d89fb617bfc"
I20250115 02:52:27.277391 5227 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:27.284725 5392 leader_election.cc:290] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [CANDIDATE]: Term 1 election: Requested vote from peers 1aacffc182d849b198c02d89fb617bfc (127.2.36.126:46751), 1b823b6b22994cc59c0ea6509ef4cf91 (127.2.36.124:42663)
I20250115 02:52:27.286248 5358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b016f34775574b05adbf5dbd2638c133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b823b6b22994cc59c0ea6509ef4cf91"
I20250115 02:52:27.286923 5358 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:27.289909 5227 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b016f34775574b05adbf5dbd2638c133 in term 1.
I20250115 02:52:27.294301 5358 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b016f34775574b05adbf5dbd2638c133 in term 1.
W20250115 02:52:27.294711 5425 catalog_manager.cc:1559] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250115 02:52:27.295393 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20250115 02:52:27.295953 5425 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:27.296022 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:52:27.295377 5266 leader_election.cc:304] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aacffc182d849b198c02d89fb617bfc, b016f34775574b05adbf5dbd2638c133; no voters:
I20250115 02:52:27.298571 5392 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 1 FOLLOWER]: Leader election won for term 1
W20250115 02:52:27.300629 5426 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:27.301071 5426 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:27.302948 5392 raft_consensus.cc:695] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 1 LEADER]: Becoming Leader. State: Replica: b016f34775574b05adbf5dbd2638c133, State: Running, Role: LEADER
I20250115 02:52:27.303571 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:27.304059 5392 consensus_queue.cc:237] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } }
W20250115 02:52:27.307772 5427 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:27.308084 5427 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:27.311717 5428 sys_catalog.cc:455] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b016f34775574b05adbf5dbd2638c133. Latest consensus state: current_term: 1 leader_uuid: "b016f34775574b05adbf5dbd2638c133" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } } }
I20250115 02:52:27.312546 5428 sys_catalog.cc:458] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:27.315179 5432 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
W20250115 02:52:27.315184 5430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:27.316754 5431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:27.321328 5434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:27.322289 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:27.322398 5432 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:52:27.323644 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:27.323930 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:27.324126 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909547324108 us; error 0 us; skew 500 ppm
I20250115 02:52:27.324909 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:27.331310 2193 webserver.cc:458] Webserver started at http://127.2.36.65:36665/ using document root <none> and password file <none>
I20250115 02:52:27.332059 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:27.332324 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:27.332677 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:27.335382 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "0b2615c5b95144a8bf7bb2411931633e"
format_stamp: "Formatted at 2025-01-15 02:52:27 on dist-test-slave-1lv2"
I20250115 02:52:27.340332 5358 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [term 1 FOLLOWER]: Refusing update from remote peer b016f34775574b05adbf5dbd2638c133: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:27.341374 5227 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [term 1 FOLLOWER]: Refusing update from remote peer b016f34775574b05adbf5dbd2638c133: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:27.342739 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250115 02:52:27.343360 5392 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:27.344965 5392 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:27.371522 5440 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:27.373605 5397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b016f34775574b05adbf5dbd2638c133. Latest consensus state: current_term: 1 leader_uuid: "b016f34775574b05adbf5dbd2638c133" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } } }
I20250115 02:52:27.374485 2193 fs_manager.cc:730] Time spent opening block manager: real 0.029s user 0.017s sys 0.011s
I20250115 02:52:27.374581 5397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:27.375018 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "0b2615c5b95144a8bf7bb2411931633e"
format_stamp: "Formatted at 2025-01-15 02:52:27 on dist-test-slave-1lv2"
I20250115 02:52:27.375463 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:27.375696 5391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b016f34775574b05adbf5dbd2638c133. Latest consensus state: current_term: 1 leader_uuid: "b016f34775574b05adbf5dbd2638c133" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } } }
I20250115 02:52:27.376488 5391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:27.379705 5428 sys_catalog.cc:455] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b016f34775574b05adbf5dbd2638c133" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } } }
I20250115 02:52:27.380638 5428 sys_catalog.cc:458] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:27.382984 5432 catalog_manager.cc:1348] Generated new cluster ID: 5da888968c7240d98bb3b823a5e55bfe
I20250115 02:52:27.383296 5432 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:52:27.386355 5397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b016f34775574b05adbf5dbd2638c133" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } } }
I20250115 02:52:27.387265 5397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:27.390137 5392 sys_catalog.cc:455] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b016f34775574b05adbf5dbd2638c133" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } } }
I20250115 02:52:27.392493 5392 sys_catalog.cc:458] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:27.413214 5391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b016f34775574b05adbf5dbd2638c133" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aacffc182d849b198c02d89fb617bfc" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46751 } } peers { permanent_uuid: "b016f34775574b05adbf5dbd2638c133" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 46189 } } peers { permanent_uuid: "1b823b6b22994cc59c0ea6509ef4cf91" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42663 } } }
I20250115 02:52:27.414088 5391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:27.417879 5432 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:52:27.420018 5432 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:52:27.445221 5432 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133: Generated new TSK 0
I20250115 02:52:27.446214 5432 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:52:27.449585 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:27.451205 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:27.452785 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:27.455289 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:27.455482 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:27.455749 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:27.455952 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:27.509399 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:37845
I20250115 02:52:27.509655 5510 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:37845 every 8 connection(s)
I20250115 02:52:27.534976 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:27.541435 5512 heartbeater.cc:346] Connected to a master server at 127.2.36.125:46189
I20250115 02:52:27.541893 5512 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:27.542783 5512 heartbeater.cc:509] Master 127.2.36.125:46189 requested a full tablet report, sending...
I20250115 02:52:27.543744 5513 heartbeater.cc:346] Connected to a master server at 127.2.36.126:46751
I20250115 02:52:27.543956 5511 heartbeater.cc:346] Connected to a master server at 127.2.36.124:42663
I20250115 02:52:27.544333 5511 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:27.545174 5511 heartbeater.cc:509] Master 127.2.36.124:42663 requested a full tablet report, sending...
I20250115 02:52:27.545687 5282 ts_manager.cc:194] Registered new tserver with Master: 0b2615c5b95144a8bf7bb2411931633e (127.2.36.65:37845)
I20250115 02:52:27.548141 5348 ts_manager.cc:194] Registered new tserver with Master: 0b2615c5b95144a8bf7bb2411931633e (127.2.36.65:37845)
I20250115 02:52:27.548344 5513 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:27.549196 5513 heartbeater.cc:509] Master 127.2.36.126:46751 requested a full tablet report, sending...
I20250115 02:52:27.548171 5282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38626
I20250115 02:52:27.552161 5217 ts_manager.cc:194] Registered new tserver with Master: 0b2615c5b95144a8bf7bb2411931633e (127.2.36.65:37845)
W20250115 02:52:27.556751 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:27.558936 5519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:27.560355 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:27.562587 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:27.563613 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:27.563843 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:27.564000 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909547563986 us; error 0 us; skew 500 ppm
I20250115 02:52:27.564504 5512 heartbeater.cc:501] Master 127.2.36.125:46189 was elected leader, sending a full tablet report...
I20250115 02:52:27.564832 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:27.570528 2193 webserver.cc:458] Webserver started at http://127.2.36.66:45509/ using document root <none> and password file <none>
I20250115 02:52:27.571066 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:27.571257 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:27.571507 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:27.572571 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "5a598b21e8b4478f9db720c291b8ccf4"
format_stamp: "Formatted at 2025-01-15 02:52:27 on dist-test-slave-1lv2"
I20250115 02:52:27.578167 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250115 02:52:27.583961 5526 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:27.584846 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250115 02:52:27.585160 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "5a598b21e8b4478f9db720c291b8ccf4"
format_stamp: "Formatted at 2025-01-15 02:52:27 on dist-test-slave-1lv2"
I20250115 02:52:27.585451 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:27.604907 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:27.606103 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:27.608539 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:27.611574 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:27.611783 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:27.612001 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:27.612154 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:27.679309 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:40671
I20250115 02:52:27.679392 5588 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:40671 every 8 connection(s)
I20250115 02:52:27.705924 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:27.710136 5589 heartbeater.cc:346] Connected to a master server at 127.2.36.124:42663
I20250115 02:52:27.710575 5589 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:27.711441 5589 heartbeater.cc:509] Master 127.2.36.124:42663 requested a full tablet report, sending...
I20250115 02:52:27.714617 5348 ts_manager.cc:194] Registered new tserver with Master: 5a598b21e8b4478f9db720c291b8ccf4 (127.2.36.66:40671)
I20250115 02:52:27.739645 5590 heartbeater.cc:346] Connected to a master server at 127.2.36.125:46189
I20250115 02:52:27.740170 5590 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:27.741173 5590 heartbeater.cc:509] Master 127.2.36.125:46189 requested a full tablet report, sending...
I20250115 02:52:27.742930 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:27.743757 5592 heartbeater.cc:346] Connected to a master server at 127.2.36.126:46751
I20250115 02:52:27.744194 5592 heartbeater.cc:463] Registering TS with master...
W20250115 02:52:27.744720 5596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:27.744769 5595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:27.745107 5592 heartbeater.cc:509] Master 127.2.36.126:46751 requested a full tablet report, sending...
W20250115 02:52:27.745047 5598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:27.746526 5282 ts_manager.cc:194] Registered new tserver with Master: 5a598b21e8b4478f9db720c291b8ccf4 (127.2.36.66:40671)
I20250115 02:52:27.749030 5282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38630
I20250115 02:52:27.746925 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:27.750144 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:27.750394 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909547750373 us; error 0 us; skew 500 ppm
I20250115 02:52:27.748291 5217 ts_manager.cc:194] Registered new tserver with Master: 5a598b21e8b4478f9db720c291b8ccf4 (127.2.36.66:40671)
I20250115 02:52:27.751313 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:27.755910 2193 webserver.cc:458] Webserver started at http://127.2.36.67:43453/ using document root <none> and password file <none>
I20250115 02:52:27.756443 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:27.756639 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:27.756875 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:27.757966 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "1b1ab4808b50412daf484a8f6ae61761"
format_stamp: "Formatted at 2025-01-15 02:52:27 on dist-test-slave-1lv2"
I20250115 02:52:27.764514 5590 heartbeater.cc:501] Master 127.2.36.125:46189 was elected leader, sending a full tablet report...
I20250115 02:52:27.769830 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.010s sys 0.003s
I20250115 02:52:27.774576 5603 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:27.776890 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250115 02:52:27.777242 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "1b1ab4808b50412daf484a8f6ae61761"
format_stamp: "Formatted at 2025-01-15 02:52:27 on dist-test-slave-1lv2"
I20250115 02:52:27.777596 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumTimeout.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:27.811070 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:27.812605 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:27.814158 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:27.817571 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:27.817836 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:27.818109 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:27.818298 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:27.956373 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:43699
I20250115 02:52:27.956452 5665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:43699 every 8 connection(s)
I20250115 02:52:27.999578 5666 heartbeater.cc:346] Connected to a master server at 127.2.36.124:42663
I20250115 02:52:28.000032 5666 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:28.000943 5666 heartbeater.cc:509] Master 127.2.36.124:42663 requested a full tablet report, sending...
I20250115 02:52:28.003679 5346 ts_manager.cc:194] Registered new tserver with Master: 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699)
I20250115 02:52:28.033497 5669 heartbeater.cc:346] Connected to a master server at 127.2.36.126:46751
I20250115 02:52:28.033946 5669 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:28.034845 5669 heartbeater.cc:509] Master 127.2.36.126:46751 requested a full tablet report, sending...
I20250115 02:52:28.037612 5215 ts_manager.cc:194] Registered new tserver with Master: 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699)
I20250115 02:52:28.040320 5667 heartbeater.cc:346] Connected to a master server at 127.2.36.125:46189
I20250115 02:52:28.040797 5667 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:28.041707 5667 heartbeater.cc:509] Master 127.2.36.125:46189 requested a full tablet report, sending...
I20250115 02:52:28.044226 5281 ts_manager.cc:194] Registered new tserver with Master: 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699)
I20250115 02:52:28.044339 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.020830437s
I20250115 02:52:28.046072 5281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38632
I20250115 02:52:28.058635 5667 heartbeater.cc:501] Master 127.2.36.125:46189 was elected leader, sending a full tablet report...
I20250115 02:52:28.128705 5281 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38636:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:52:28.130896 5281 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:52:28.314083 5427 catalog_manager.cc:1260] Loaded cluster ID: 5da888968c7240d98bb3b823a5e55bfe
I20250115 02:52:28.314463 5427 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91: loading cluster ID for follower catalog manager: success
I20250115 02:52:28.319290 5426 catalog_manager.cc:1260] Loaded cluster ID: 5da888968c7240d98bb3b823a5e55bfe
I20250115 02:52:28.319649 5426 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc: loading cluster ID for follower catalog manager: success
I20250115 02:52:28.366140 5427 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91: acquiring CA information for follower catalog manager: success
I20250115 02:52:28.378180 5426 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc: acquiring CA information for follower catalog manager: success
I20250115 02:52:28.400317 5426 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:28.420733 5554 tablet_service.cc:1467] Processing CreateTablet for tablet a7d1cb5010574224984a249f01ad377d (DEFAULT_TABLE table=ksck-test-table [id=65abd001b2064a63abf42e0371d442e1]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:28.422278 5554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7d1cb5010574224984a249f01ad377d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:28.428932 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab11ff0056843918d39548dc3ab9197 (DEFAULT_TABLE table=ksck-test-table [id=65abd001b2064a63abf42e0371d442e1]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:28.430327 5553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ab11ff0056843918d39548dc3ab9197. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:28.451807 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 0967b4bfc16a458c99fc755770762dc3 (DEFAULT_TABLE table=ksck-test-table [id=65abd001b2064a63abf42e0371d442e1]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:28.453219 5552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0967b4bfc16a458c99fc755770762dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:28.472795 5630 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab11ff0056843918d39548dc3ab9197 (DEFAULT_TABLE table=ksck-test-table [id=65abd001b2064a63abf42e0371d442e1]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:28.473067 5631 tablet_service.cc:1467] Processing CreateTablet for tablet a7d1cb5010574224984a249f01ad377d (DEFAULT_TABLE table=ksck-test-table [id=65abd001b2064a63abf42e0371d442e1]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:28.474293 5631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7d1cb5010574224984a249f01ad377d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:28.474434 5629 tablet_service.cc:1467] Processing CreateTablet for tablet 0967b4bfc16a458c99fc755770762dc3 (DEFAULT_TABLE table=ksck-test-table [id=65abd001b2064a63abf42e0371d442e1]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:28.475693 5629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0967b4bfc16a458c99fc755770762dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:28.481666 5476 tablet_service.cc:1467] Processing CreateTablet for tablet a7d1cb5010574224984a249f01ad377d (DEFAULT_TABLE table=ksck-test-table [id=65abd001b2064a63abf42e0371d442e1]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:28.482996 5476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7d1cb5010574224984a249f01ad377d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:28.485718 5630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ab11ff0056843918d39548dc3ab9197. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:28.488359 5475 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab11ff0056843918d39548dc3ab9197 (DEFAULT_TABLE table=ksck-test-table [id=65abd001b2064a63abf42e0371d442e1]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:28.489339 5475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ab11ff0056843918d39548dc3ab9197. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:28.511826 5474 tablet_service.cc:1467] Processing CreateTablet for tablet 0967b4bfc16a458c99fc755770762dc3 (DEFAULT_TABLE table=ksck-test-table [id=65abd001b2064a63abf42e0371d442e1]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:28.515113 5427 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:28.518133 5474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0967b4bfc16a458c99fc755770762dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:28.550441 5695 tablet_bootstrap.cc:492] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4: Bootstrap starting.
I20250115 02:52:28.550338 5694 tablet_bootstrap.cc:492] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761: Bootstrap starting.
I20250115 02:52:28.556435 5695 tablet_bootstrap.cc:654] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:28.575251 5695 tablet_bootstrap.cc:492] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4: No bootstrap required, opened a new log
I20250115 02:52:28.577351 5694 tablet_bootstrap.cc:654] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:28.579356 5695 ts_tablet_manager.cc:1397] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.000s
I20250115 02:52:28.582376 5695 raft_consensus.cc:357] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.583216 5695 raft_consensus.cc:383] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:28.583516 5695 raft_consensus.cc:738] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a598b21e8b4478f9db720c291b8ccf4, State: Initialized, Role: FOLLOWER
I20250115 02:52:28.588059 5695 consensus_queue.cc:260] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.590655 5696 tablet_bootstrap.cc:492] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e: Bootstrap starting.
I20250115 02:52:28.600068 5695 ts_tablet_manager.cc:1428] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20250115 02:52:28.604084 5696 tablet_bootstrap.cc:654] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:28.611817 5695 tablet_bootstrap.cc:492] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4: Bootstrap starting.
I20250115 02:52:28.618166 5695 tablet_bootstrap.cc:654] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:28.620822 5699 raft_consensus.cc:491] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:28.622853 5694 tablet_bootstrap.cc:492] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761: No bootstrap required, opened a new log
I20250115 02:52:28.624794 5694 ts_tablet_manager.cc:1397] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.013s
I20250115 02:52:28.623821 5696 tablet_bootstrap.cc:492] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e: No bootstrap required, opened a new log
I20250115 02:52:28.625634 5696 ts_tablet_manager.cc:1397] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.003s
I20250115 02:52:28.628134 5694 raft_consensus.cc:357] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.631084 5696 raft_consensus.cc:357] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } }
I20250115 02:52:28.632930 5695 tablet_bootstrap.cc:492] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4: No bootstrap required, opened a new log
I20250115 02:52:28.639329 5696 raft_consensus.cc:383] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:28.639603 5695 ts_tablet_manager.cc:1397] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.004s
I20250115 02:52:28.639715 5696 raft_consensus.cc:738] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b2615c5b95144a8bf7bb2411931633e, State: Initialized, Role: FOLLOWER
I20250115 02:52:28.639439 5694 raft_consensus.cc:383] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:28.640527 5694 raft_consensus.cc:738] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1ab4808b50412daf484a8f6ae61761, State: Initialized, Role: FOLLOWER
I20250115 02:52:28.640606 5696 consensus_queue.cc:260] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } }
I20250115 02:52:28.641989 5694 consensus_queue.cc:260] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.642665 5695 raft_consensus.cc:357] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } }
I20250115 02:52:28.644158 5695 raft_consensus.cc:383] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:28.644502 5695 raft_consensus.cc:738] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a598b21e8b4478f9db720c291b8ccf4, State: Initialized, Role: FOLLOWER
I20250115 02:52:28.645385 5695 consensus_queue.cc:260] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } }
I20250115 02:52:28.647795 5695 ts_tablet_manager.cc:1428] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20250115 02:52:28.648873 5695 tablet_bootstrap.cc:492] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4: Bootstrap starting.
I20250115 02:52:28.651357 5696 ts_tablet_manager.cc:1428] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e: Time spent starting tablet: real 0.025s user 0.008s sys 0.003s
I20250115 02:52:28.652498 5696 tablet_bootstrap.cc:492] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e: Bootstrap starting.
I20250115 02:52:28.623883 5699 raft_consensus.cc:513] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.658772 5696 tablet_bootstrap.cc:654] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:28.663154 5695 tablet_bootstrap.cc:654] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:28.664927 5696 tablet_bootstrap.cc:492] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e: No bootstrap required, opened a new log
I20250115 02:52:28.665449 5696 ts_tablet_manager.cc:1397] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.001s
I20250115 02:52:28.666874 5694 ts_tablet_manager.cc:1428] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761: Time spent starting tablet: real 0.041s user 0.004s sys 0.006s
I20250115 02:52:28.668246 5696 raft_consensus.cc:357] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.669698 5699 leader_election.cc:290] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b2615c5b95144a8bf7bb2411931633e (127.2.36.65:37845), 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699)
I20250115 02:52:28.673470 5696 raft_consensus.cc:383] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:28.673843 5696 raft_consensus.cc:738] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b2615c5b95144a8bf7bb2411931633e, State: Initialized, Role: FOLLOWER
I20250115 02:52:28.674953 5696 consensus_queue.cc:260] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.679203 5695 tablet_bootstrap.cc:492] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4: No bootstrap required, opened a new log
I20250115 02:52:28.679802 5695 ts_tablet_manager.cc:1397] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.004s
I20250115 02:52:28.680244 5694 tablet_bootstrap.cc:492] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761: Bootstrap starting.
I20250115 02:52:28.682727 5695 raft_consensus.cc:357] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.683521 5695 raft_consensus.cc:383] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:28.683815 5695 raft_consensus.cc:738] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a598b21e8b4478f9db720c291b8ccf4, State: Initialized, Role: FOLLOWER
I20250115 02:52:28.687007 5696 ts_tablet_manager.cc:1428] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e: Time spent starting tablet: real 0.021s user 0.008s sys 0.002s
I20250115 02:52:28.687651 5695 consensus_queue.cc:260] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.692706 5696 tablet_bootstrap.cc:492] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e: Bootstrap starting.
I20250115 02:52:28.696705 5694 tablet_bootstrap.cc:654] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:28.698664 5695 ts_tablet_manager.cc:1428] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4: Time spent starting tablet: real 0.019s user 0.002s sys 0.004s
I20250115 02:52:28.699281 5696 tablet_bootstrap.cc:654] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:28.715715 5703 raft_consensus.cc:491] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:28.716241 5703 raft_consensus.cc:513] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.720073 5703 leader_election.cc:290] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a598b21e8b4478f9db720c291b8ccf4 (127.2.36.66:40671), 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699)
I20250115 02:52:28.770555 5564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab11ff0056843918d39548dc3ab9197" candidate_uuid: "0b2615c5b95144a8bf7bb2411931633e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a598b21e8b4478f9db720c291b8ccf4" is_pre_election: true
I20250115 02:52:28.771311 5564 raft_consensus.cc:2463] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b2615c5b95144a8bf7bb2411931633e in term 0.
I20250115 02:52:28.772940 5448 leader_election.cc:304] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b2615c5b95144a8bf7bb2411931633e, 5a598b21e8b4478f9db720c291b8ccf4; no voters:
I20250115 02:52:28.773854 5703 raft_consensus.cc:2798] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:28.774200 5703 raft_consensus.cc:491] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:28.774502 5703 raft_consensus.cc:3054] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:28.779930 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab11ff0056843918d39548dc3ab9197" candidate_uuid: "5a598b21e8b4478f9db720c291b8ccf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1ab4808b50412daf484a8f6ae61761" is_pre_election: true
I20250115 02:52:28.784811 5703 raft_consensus.cc:513] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.791237 5703 leader_election.cc:290] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [CANDIDATE]: Term 1 election: Requested vote from peers 5a598b21e8b4478f9db720c291b8ccf4 (127.2.36.66:40671), 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699)
I20250115 02:52:28.795352 5564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab11ff0056843918d39548dc3ab9197" candidate_uuid: "0b2615c5b95144a8bf7bb2411931633e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a598b21e8b4478f9db720c291b8ccf4"
I20250115 02:52:28.796063 5564 raft_consensus.cc:3054] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:28.801429 5694 tablet_bootstrap.cc:492] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761: No bootstrap required, opened a new log
I20250115 02:52:28.802084 5694 ts_tablet_manager.cc:1397] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761: Time spent bootstrapping tablet: real 0.123s user 0.006s sys 0.008s
W20250115 02:52:28.803367 5527 leader_election.cc:343] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:52:28.804042 5700 raft_consensus.cc:491] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:28.804646 5700 raft_consensus.cc:513] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.807760 5564 raft_consensus.cc:2463] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b2615c5b95144a8bf7bb2411931633e in term 1.
I20250115 02:52:28.809494 5700 leader_election.cc:290] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b2615c5b95144a8bf7bb2411931633e (127.2.36.65:37845), 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699)
I20250115 02:52:28.811411 5448 leader_election.cc:304] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b2615c5b95144a8bf7bb2411931633e, 5a598b21e8b4478f9db720c291b8ccf4; no voters:
I20250115 02:52:28.813015 5703 raft_consensus.cc:2798] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:28.815256 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab11ff0056843918d39548dc3ab9197" candidate_uuid: "0b2615c5b95144a8bf7bb2411931633e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1ab4808b50412daf484a8f6ae61761" is_pre_election: true
I20250115 02:52:28.816228 5639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab11ff0056843918d39548dc3ab9197" candidate_uuid: "0b2615c5b95144a8bf7bb2411931633e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1ab4808b50412daf484a8f6ae61761"
W20250115 02:52:28.817479 5446 leader_election.cc:343] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699): Illegal state: must be running to vote when last-logged opid is not known
W20250115 02:52:28.818471 5446 leader_election.cc:343] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:52:28.816563 5640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0967b4bfc16a458c99fc755770762dc3" candidate_uuid: "5a598b21e8b4478f9db720c291b8ccf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1ab4808b50412daf484a8f6ae61761" is_pre_election: true
I20250115 02:52:28.819705 5640 raft_consensus.cc:2463] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a598b21e8b4478f9db720c291b8ccf4 in term 0.
I20250115 02:52:28.823302 5527 leader_election.cc:304] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1ab4808b50412daf484a8f6ae61761, 5a598b21e8b4478f9db720c291b8ccf4; no voters:
I20250115 02:52:28.823453 5485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0967b4bfc16a458c99fc755770762dc3" candidate_uuid: "5a598b21e8b4478f9db720c291b8ccf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b2615c5b95144a8bf7bb2411931633e" is_pre_election: true
I20250115 02:52:28.825033 5486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab11ff0056843918d39548dc3ab9197" candidate_uuid: "5a598b21e8b4478f9db720c291b8ccf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b2615c5b95144a8bf7bb2411931633e" is_pre_election: true
W20250115 02:52:28.825237 5527 leader_election.cc:343] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b2615c5b95144a8bf7bb2411931633e (127.2.36.65:37845): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:52:28.825146 5694 raft_consensus.cc:357] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } }
I20250115 02:52:28.825942 5694 raft_consensus.cc:383] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:28.826418 5694 raft_consensus.cc:738] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1ab4808b50412daf484a8f6ae61761, State: Initialized, Role: FOLLOWER
I20250115 02:52:28.827427 5700 raft_consensus.cc:2798] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:28.827142 5694 consensus_queue.cc:260] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } }
I20250115 02:52:28.827816 5700 raft_consensus.cc:491] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:28.828190 5700 raft_consensus.cc:3054] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:28.829499 5694 ts_tablet_manager.cc:1428] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761: Time spent starting tablet: real 0.027s user 0.003s sys 0.003s
I20250115 02:52:28.830526 5694 tablet_bootstrap.cc:492] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761: Bootstrap starting.
I20250115 02:52:28.833999 5696 tablet_bootstrap.cc:492] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e: No bootstrap required, opened a new log
I20250115 02:52:28.834542 5696 ts_tablet_manager.cc:1397] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e: Time spent bootstrapping tablet: real 0.142s user 0.032s sys 0.035s
I20250115 02:52:28.836803 5700 raft_consensus.cc:513] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.837335 5696 raft_consensus.cc:357] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.838064 5696 raft_consensus.cc:383] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:28.838363 5696 raft_consensus.cc:738] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b2615c5b95144a8bf7bb2411931633e, State: Initialized, Role: FOLLOWER
I20250115 02:52:28.824427 5703 raft_consensus.cc:695] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 1 LEADER]: Becoming Leader. State: Replica: 0b2615c5b95144a8bf7bb2411931633e, State: Running, Role: LEADER
I20250115 02:52:28.839565 5703 consensus_queue.cc:237] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.842489 5485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0967b4bfc16a458c99fc755770762dc3" candidate_uuid: "5a598b21e8b4478f9db720c291b8ccf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b2615c5b95144a8bf7bb2411931633e"
I20250115 02:52:28.843469 5696 consensus_queue.cc:260] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.844542 5485 raft_consensus.cc:3054] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:28.850683 5700 leader_election.cc:290] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [CANDIDATE]: Term 1 election: Requested vote from peers 0b2615c5b95144a8bf7bb2411931633e (127.2.36.65:37845), 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699)
I20250115 02:52:28.857012 5694 tablet_bootstrap.cc:654] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:28.859580 5485 raft_consensus.cc:2463] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a598b21e8b4478f9db720c291b8ccf4 in term 1.
I20250115 02:52:28.863781 5640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0967b4bfc16a458c99fc755770762dc3" candidate_uuid: "5a598b21e8b4478f9db720c291b8ccf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1ab4808b50412daf484a8f6ae61761"
I20250115 02:52:28.865029 5696 ts_tablet_manager.cc:1428] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e: Time spent starting tablet: real 0.030s user 0.007s sys 0.002s
I20250115 02:52:28.864513 5640 raft_consensus.cc:3054] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:28.874509 5640 raft_consensus.cc:2463] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a598b21e8b4478f9db720c291b8ccf4 in term 1.
I20250115 02:52:28.878007 5694 tablet_bootstrap.cc:492] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761: No bootstrap required, opened a new log
I20250115 02:52:28.878723 5694 ts_tablet_manager.cc:1397] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.003s
I20250115 02:52:28.881129 5694 raft_consensus.cc:357] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.881908 5694 raft_consensus.cc:383] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:28.882192 5694 raft_consensus.cc:738] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1ab4808b50412daf484a8f6ae61761, State: Initialized, Role: FOLLOWER
I20250115 02:52:28.882941 5694 consensus_queue.cc:260] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.886119 5694 ts_tablet_manager.cc:1428] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20250115 02:52:28.889226 5281 catalog_manager.cc:5526] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b2615c5b95144a8bf7bb2411931633e (127.2.36.65). New cstate: current_term: 1 leader_uuid: "0b2615c5b95144a8bf7bb2411931633e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:28.901249 5527 leader_election.cc:304] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a598b21e8b4478f9db720c291b8ccf4; no voters: 0b2615c5b95144a8bf7bb2411931633e, 1b1ab4808b50412daf484a8f6ae61761
I20250115 02:52:28.902199 5700 raft_consensus.cc:2743] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:52:28.905619 5527 leader_election.cc:304] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b2615c5b95144a8bf7bb2411931633e, 5a598b21e8b4478f9db720c291b8ccf4; no voters:
I20250115 02:52:28.907362 5700 raft_consensus.cc:2798] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:28.907872 5700 raft_consensus.cc:695] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 LEADER]: Becoming Leader. State: Replica: 5a598b21e8b4478f9db720c291b8ccf4, State: Running, Role: LEADER
I20250115 02:52:28.908686 5700 consensus_queue.cc:237] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:28.969552 5278 catalog_manager.cc:5526] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a598b21e8b4478f9db720c291b8ccf4 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "5a598b21e8b4478f9db720c291b8ccf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:28.972167 5715 raft_consensus.cc:491] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:28.972671 5715 raft_consensus.cc:513] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } }
I20250115 02:52:28.974720 5715 leader_election.cc:290] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699), 5a598b21e8b4478f9db720c291b8ccf4 (127.2.36.66:40671)
I20250115 02:52:28.975674 5640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d1cb5010574224984a249f01ad377d" candidate_uuid: "0b2615c5b95144a8bf7bb2411931633e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1ab4808b50412daf484a8f6ae61761" is_pre_election: true
I20250115 02:52:28.976385 5640 raft_consensus.cc:2463] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b2615c5b95144a8bf7bb2411931633e in term 0.
I20250115 02:52:28.976418 5564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d1cb5010574224984a249f01ad377d" candidate_uuid: "0b2615c5b95144a8bf7bb2411931633e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a598b21e8b4478f9db720c291b8ccf4" is_pre_election: true
I20250115 02:52:28.977370 5564 raft_consensus.cc:2463] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b2615c5b95144a8bf7bb2411931633e in term 0.
I20250115 02:52:28.977882 5446 leader_election.cc:304] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b2615c5b95144a8bf7bb2411931633e, 1b1ab4808b50412daf484a8f6ae61761; no voters:
I20250115 02:52:28.978700 5715 raft_consensus.cc:2798] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:28.979095 5715 raft_consensus.cc:491] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:28.979401 5715 raft_consensus.cc:3054] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:28.986124 5715 raft_consensus.cc:513] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } }
I20250115 02:52:28.989928 5640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d1cb5010574224984a249f01ad377d" candidate_uuid: "0b2615c5b95144a8bf7bb2411931633e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1ab4808b50412daf484a8f6ae61761"
I20250115 02:52:28.990542 5640 raft_consensus.cc:3054] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:29.006224 5640 raft_consensus.cc:2463] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b2615c5b95144a8bf7bb2411931633e in term 1.
I20250115 02:52:29.007381 5446 leader_election.cc:304] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b2615c5b95144a8bf7bb2411931633e, 1b1ab4808b50412daf484a8f6ae61761; no voters:
I20250115 02:52:29.008230 5703 raft_consensus.cc:2798] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:29.008728 5703 raft_consensus.cc:695] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 1 LEADER]: Becoming Leader. State: Replica: 0b2615c5b95144a8bf7bb2411931633e, State: Running, Role: LEADER
I20250115 02:52:29.009538 5703 consensus_queue.cc:237] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } }
I20250115 02:52:29.023386 5564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d1cb5010574224984a249f01ad377d" candidate_uuid: "0b2615c5b95144a8bf7bb2411931633e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a598b21e8b4478f9db720c291b8ccf4"
I20250115 02:52:29.024084 5564 raft_consensus.cc:3054] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:29.026548 5715 leader_election.cc:290] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [CANDIDATE]: Term 1 election: Requested vote from peers 1b1ab4808b50412daf484a8f6ae61761 (127.2.36.67:43699), 5a598b21e8b4478f9db720c291b8ccf4 (127.2.36.66:40671)
I20250115 02:52:29.045120 5564 raft_consensus.cc:2463] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b2615c5b95144a8bf7bb2411931633e in term 1.
I20250115 02:52:29.045579 5281 catalog_manager.cc:5526] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b2615c5b95144a8bf7bb2411931633e (127.2.36.65). New cstate: current_term: 1 leader_uuid: "0b2615c5b95144a8bf7bb2411931633e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:29.095177 5704 raft_consensus.cc:491] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:29.095679 5704 raft_consensus.cc:513] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 } } peers { permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 } } peers { permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 } }
I20250115 02:52:29.097889 5704 leader_election.cc:290] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b2615c5b95144a8bf7bb2411931633e (127.2.36.65:37845), 5a598b21e8b4478f9db720c291b8ccf4 (127.2.36.66:40671)
I20250115 02:52:29.128085 5564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab11ff0056843918d39548dc3ab9197" candidate_uuid: "1b1ab4808b50412daf484a8f6ae61761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a598b21e8b4478f9db720c291b8ccf4" is_pre_election: true
I20250115 02:52:29.128971 5564 raft_consensus.cc:2388] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b1ab4808b50412daf484a8f6ae61761 in current term 1: Already voted for candidate 0b2615c5b95144a8bf7bb2411931633e in this term.
I20250115 02:52:29.130584 5485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab11ff0056843918d39548dc3ab9197" candidate_uuid: "1b1ab4808b50412daf484a8f6ae61761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b2615c5b95144a8bf7bb2411931633e" is_pre_election: true
I20250115 02:52:29.131856 5604 leader_election.cc:304] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b1ab4808b50412daf484a8f6ae61761; no voters: 0b2615c5b95144a8bf7bb2411931633e, 5a598b21e8b4478f9db720c291b8ccf4
I20250115 02:52:29.132591 5704 raft_consensus.cc:3054] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:29.138325 5704 raft_consensus.cc:2743] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:52:29.264931 2193 ksck_remote-test.cc:410] Generating row writes...
I20250115 02:52:29.364960 5700 consensus_queue.cc:1035] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:29.415745 5700 consensus_queue.cc:1035] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b2615c5b95144a8bf7bb2411931633e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 37845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:29.444633 5703 consensus_queue.cc:1035] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:29.482821 5714 consensus_queue.cc:1035] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:29.501011 5714 consensus_queue.cc:1035] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b1ab4808b50412daf484a8f6ae61761" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:29.512792 5714 consensus_queue.cc:1035] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a598b21e8b4478f9db720c291b8ccf4" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 02:52:31.200099 5763 log.cc:926] Time spent T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4: Append to log took a long time: real 0.107s user 0.014s sys 0.000s
W20250115 02:52:31.232932 5767 log.cc:926] Time spent T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e: Append to log took a long time: real 0.082s user 0.013s sys 0.000s
W20250115 02:52:32.348733 5774 log.cc:926] Time spent T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4: Append to log took a long time: real 0.070s user 0.013s sys 0.000s
I20250115 02:52:33.555722 5756 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51052 (ReqId={client: 7aa4c8732ce24432810418e5c0848e7c, seq_no=1, attempt_no=0}) took 2900 ms. Trace:
I20250115 02:52:33.563058 5756 rpcz_store.cc:276] 0115 02:52:30.655391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:52:30.657088 (+ 1697us) service_pool.cc:225] Handling call
0115 02:52:33.555536 (+2898448us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0115 02:52:30.665365 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0967b4bfc16a458c99fc755770762dc3
0115 02:52:30.665708 (+ 343us) write_op.cc:432] Acquiring schema lock in shared mode
0115 02:52:30.665731 (+ 23us) write_op.cc:435] Acquired schema lock
0115 02:52:30.665747 (+ 16us) tablet.cc:659] Decoding operations
0115 02:52:30.851810 (+186063us) write_op.cc:620] Acquiring the partition lock for write op
0115 02:52:30.851860 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0115 02:52:30.851884 (+ 24us) tablet.cc:682] Acquiring locks for 4243 operations
0115 02:52:31.003882 (+151998us) tablet.cc:698] Row locks acquired
0115 02:52:31.003901 (+ 19us) write_op.cc:260] PREPARE: finished
0115 02:52:31.004073 (+ 172us) write_op.cc:270] Start()
0115 02:52:31.004185 (+ 112us) write_op.cc:276] Timestamp: P: 1736909551004027 usec, L: 0
0115 02:52:31.004205 (+ 20us) op_driver.cc:348] REPLICATION: starting
0115 02:52:31.005138 (+ 933us) log.cc:844] Serialized 42717 byte log entry
0115 02:52:31.201335 (+196197us) op_driver.cc:464] REPLICATION: finished
0115 02:52:31.202478 (+ 1143us) write_op.cc:301] APPLY: starting
0115 02:52:31.202552 (+ 74us) tablet.cc:1364] starting BulkCheckPresence
0115 02:52:31.464014 (+261462us) tablet.cc:1367] finished BulkCheckPresence
0115 02:52:31.464871 (+ 857us) tablet.cc:1369] starting ApplyRowOperation cycle
0115 02:52:33.027788 (+1562917us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b5c000d890c. stack: 00007f5a18a69f63 00007f5a18a69e60 00007f5a183bb584 00007f5a2242cff4 00007f5a2242cf89 00007f5a21f67521 00007f5a21f67411 00007f5a21f6214c 00007f5a1f30b02b 00007f5a1f30acf3 00007f5a1f408c24 00007f5a1f3f47ac 00007f5a1f3f8b91 00007f5a1f3f8b49 00007f5a1f3f8ad9 00007f5a1f3f8aa1
0115 02:52:33.048976 (+ 21188us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b5c000d890c. stack: 00007f5a18a69f63 00007f5a18a69e60 00007f5a183bb584 00007f5a2242cff4 00007f5a2242cf89 00007f5a21f67521 00007f5a21f67411 00007f5a21f6214c 00007f5a1f30b02b 00007f5a1f30acf3 00007f5a1f408c24 00007f5a1f3f47ac 00007f5a1f3f8b91 00007f5a1f3f8b49 00007f5a1f3f8ad9 00007f5a1f3f8aa1
0115 02:52:33.379782 (+330806us) tablet.cc:1380] finished ApplyRowOperation cycle
0115 02:52:33.418446 (+ 38664us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0115 02:52:33.418486 (+ 40us) write_op.cc:312] APPLY: finished
0115 02:52:33.496661 (+ 78175us) log.cc:844] Serialized 25480 byte log entry
0115 02:52:33.513726 (+ 17065us) write_op.cc:489] Releasing partition, row and schema locks
0115 02:52:33.552935 (+ 39209us) write_op.cc:454] Released schema lock
0115 02:52:33.555283 (+ 2348us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":956,"mutex_wait_us":63,"num_ops":4243,"prepare.queue_time_us":6207,"prepare.run_cpu_time_us":397670,"prepare.run_wall_time_us":490191,"raft.queue_time_us":1972,"raft.run_cpu_time_us":3740,"raft.run_wall_time_us":16879,"replication_time_us":197002,"spinlock_wait_cycles":40738304,"thread_start_us":195878,"threads_started":4,"wal-append.queue_time_us":103157,"wal-append.run_cpu_time_us":16254,"wal-append.run_wall_time_us":1112798}]]}
I20250115 02:52:33.774313 5777 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51052 (ReqId={client: 7aa4c8732ce24432810418e5c0848e7c, seq_no=3, attempt_no=0}) took 1920 ms. Trace:
I20250115 02:52:33.774860 5777 rpcz_store.cc:276] 0115 02:52:31.854270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:52:31.854438 (+ 168us) service_pool.cc:225] Handling call
0115 02:52:33.774250 (+1919812us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0115 02:52:31.883504 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 0967b4bfc16a458c99fc755770762dc3
0115 02:52:31.885327 (+ 1823us) write_op.cc:432] Acquiring schema lock in shared mode
0115 02:52:31.885360 (+ 33us) write_op.cc:435] Acquired schema lock
0115 02:52:31.885376 (+ 16us) tablet.cc:659] Decoding operations
0115 02:52:32.055047 (+169671us) write_op.cc:620] Acquiring the partition lock for write op
0115 02:52:32.055099 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0115 02:52:32.055122 (+ 23us) tablet.cc:682] Acquiring locks for 4660 operations
0115 02:52:32.269711 (+214589us) tablet.cc:698] Row locks acquired
0115 02:52:32.269730 (+ 19us) write_op.cc:260] PREPARE: finished
0115 02:52:32.269906 (+ 176us) write_op.cc:270] Start()
0115 02:52:32.270010 (+ 104us) write_op.cc:276] Timestamp: P: 1736909552269863 usec, L: 0
0115 02:52:32.270028 (+ 18us) op_driver.cc:348] REPLICATION: starting
0115 02:52:32.270901 (+ 873us) log.cc:844] Serialized 46896 byte log entry
0115 02:52:32.353323 (+ 82422us) op_driver.cc:464] REPLICATION: finished
0115 02:52:32.378153 (+ 24830us) write_op.cc:301] APPLY: starting
0115 02:52:32.378230 (+ 77us) tablet.cc:1364] starting BulkCheckPresence
0115 02:52:32.698045 (+319815us) tablet.cc:1367] finished BulkCheckPresence
0115 02:52:32.698153 (+ 108us) tablet.cc:1369] starting ApplyRowOperation cycle
0115 02:52:32.845865 (+147712us) spinlock_profiling.cc:243] Waited 8.77 ms on lock 0x7b5c000d890c. stack: 00007f5a18a69f63 00007f5a18a69e60 00007f5a183bb584 00007f5a2242cff4 00007f5a2242cf89 00007f5a21f67521 00007f5a21f67411 00007f5a21f6214c 00007f5a1f30b02b 00007f5a1f30acf3 00007f5a1f408c24 00007f5a1f3f47ac 00007f5a1f3f8b91 00007f5a1f3f8b49 00007f5a1f3f8ad9 00007f5a1f3f8aa1
0115 02:52:33.707235 (+861370us) tablet.cc:1380] finished ApplyRowOperation cycle
0115 02:52:33.725423 (+ 18188us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0115 02:52:33.725462 (+ 39us) write_op.cc:312] APPLY: finished
0115 02:52:33.757270 (+ 31808us) log.cc:844] Serialized 27982 byte log entry
0115 02:52:33.757453 (+ 183us) write_op.cc:489] Releasing partition, row and schema locks
0115 02:52:33.771421 (+ 13968us) write_op.cc:454] Released schema lock
0115 02:52:33.773973 (+ 2552us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":24596,"num_ops":4660,"prepare.queue_time_us":27111,"prepare.run_cpu_time_us":299128,"prepare.run_wall_time_us":419266,"raft.queue_time_us":270,"raft.run_cpu_time_us":1976,"raft.run_wall_time_us":1977,"replication_time_us":83157,"spinlock_wait_cycles":20295168,"thread_start_us":61511,"threads_started":3,"wal-append.queue_time_us":7519,"wal-append.run_cpu_time_us":16351,"wal-append.run_wall_time_us":1071420}]]}
W20250115 02:52:33.967808 5675 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.2.36.66:40671, user_credentials={real_user=slave}} blocked reactor thread for 391708us
W20250115 02:52:34.174391 5675 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.2.36.66:40671, user_credentials={real_user=slave}} blocked reactor thread for 205555us
I20250115 02:52:34.979530 5476 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250115 02:52:34.988688 5552 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:52:34.990907 5630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:52:35.369143 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
W20250115 02:52:35.370245 5732 ksck_checksum.cc:286] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down.
W20250115 02:52:35.390192 5731 ksck_checksum.cc:286] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down.
W20250115 02:52:35.390995 5731 ksck_checksum.cc:286] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down.
W20250115 02:52:35.391764 5731 ksck_checksum.cc:286] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down.
W20250115 02:52:35.392500 5731 ksck_checksum.cc:286] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down.
I20250115 02:52:35.456792 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:35.458110 2193 tablet_replica.cc:331] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e: stopping tablet replica
I20250115 02:52:35.458981 2193 raft_consensus.cc:2238] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:35.459573 2193 raft_consensus.cc:2267] T 0967b4bfc16a458c99fc755770762dc3 P 0b2615c5b95144a8bf7bb2411931633e [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 02:52:35.493341 5527 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 7 similar messages]
W20250115 02:52:35.496569 5527 consensus_peers.cc:487] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 -> Peer 0b2615c5b95144a8bf7bb2411931633e (127.2.36.65:37845): Couldn't send request to peer 0b2615c5b95144a8bf7bb2411931633e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:35.514397 2193 tablet_replica.cc:331] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e: stopping tablet replica
I20250115 02:52:35.515053 2193 raft_consensus.cc:2238] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:35.515928 2193 raft_consensus.cc:2267] T 3ab11ff0056843918d39548dc3ab9197 P 0b2615c5b95144a8bf7bb2411931633e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:35.517753 2193 tablet_replica.cc:331] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e: stopping tablet replica
I20250115 02:52:35.518224 2193 raft_consensus.cc:2238] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:35.519259 2193 raft_consensus.cc:2267] T a7d1cb5010574224984a249f01ad377d P 0b2615c5b95144a8bf7bb2411931633e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:35.554035 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:52:35.580060 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
W20250115 02:52:35.631333 5731 ksck_checksum.cc:286] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down.
W20250115 02:52:35.638705 5732 ksck_checksum.cc:286] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down.
I20250115 02:52:35.646057 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:35.646646 2193 tablet_replica.cc:331] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4: stopping tablet replica
I20250115 02:52:35.647303 2193 raft_consensus.cc:2238] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:35.648000 2193 raft_consensus.cc:2267] T 0967b4bfc16a458c99fc755770762dc3 P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:35.661432 2193 tablet_replica.cc:331] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4: stopping tablet replica
I20250115 02:52:35.661940 2193 raft_consensus.cc:2238] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:35.662323 2193 raft_consensus.cc:2267] T a7d1cb5010574224984a249f01ad377d P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:35.664356 2193 tablet_replica.cc:331] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4: stopping tablet replica
I20250115 02:52:35.664798 2193 raft_consensus.cc:2238] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:35.665184 2193 raft_consensus.cc:2267] T 3ab11ff0056843918d39548dc3ab9197 P 5a598b21e8b4478f9db720c291b8ccf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:35.687302 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:52:35.701104 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:52:35.727093 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:35.727648 2193 tablet_replica.cc:331] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761: stopping tablet replica
I20250115 02:52:35.728168 2193 raft_consensus.cc:2238] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:35.728598 2193 raft_consensus.cc:2267] T 3ab11ff0056843918d39548dc3ab9197 P 1b1ab4808b50412daf484a8f6ae61761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:35.730250 2193 tablet_replica.cc:331] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761: stopping tablet replica
I20250115 02:52:35.730708 2193 raft_consensus.cc:2238] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:35.731144 2193 raft_consensus.cc:2267] T a7d1cb5010574224984a249f01ad377d P 1b1ab4808b50412daf484a8f6ae61761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:35.732697 2193 tablet_replica.cc:331] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761: stopping tablet replica
I20250115 02:52:35.733110 2193 raft_consensus.cc:2238] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:35.733493 2193 raft_consensus.cc:2267] T 0967b4bfc16a458c99fc755770762dc3 P 1b1ab4808b50412daf484a8f6ae61761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:35.761538 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:52:35.773913 2193 master.cc:537] Master@127.2.36.126:46751 shutting down...
I20250115 02:52:35.790647 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:35.791165 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:35.791414 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 1aacffc182d849b198c02d89fb617bfc: stopping tablet replica
I20250115 02:52:35.809899 2193 master.cc:559] Master@127.2.36.126:46751 shutdown complete.
I20250115 02:52:35.820658 2193 master.cc:537] Master@127.2.36.125:46189 shutting down...
I20250115 02:52:35.834700 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:35.835510 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:35.835801 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P b016f34775574b05adbf5dbd2638c133: stopping tablet replica
I20250115 02:52:35.855086 2193 master.cc:559] Master@127.2.36.125:46189 shutdown complete.
I20250115 02:52:35.866379 2193 master.cc:537] Master@127.2.36.124:42663 shutting down...
I20250115 02:52:35.880396 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:35.880887 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:35.881181 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 1b823b6b22994cc59c0ea6509ef4cf91: stopping tablet replica
I20250115 02:52:35.899324 2193 master.cc:559] Master@127.2.36.124:42663 shutdown complete.
[ OK ] RemoteKsckTest.TestChecksumTimeout (9187 ms)
[ RUN ] RemoteKsckTest.TestChecksumSnapshot
I20250115 02:52:35.928779 2193 test_util.cc:274] Using random seed: -988377419
I20250115 02:52:35.929690 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:39537,127.2.36.125:40647,127.2.36.124:34495
I20250115 02:52:35.930908 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:35.936558 5818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:35.937158 5819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:35.938374 5821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:35.938932 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:35.942030 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:35.942211 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:35.942345 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909555942328 us; error 0 us; skew 500 ppm
I20250115 02:52:35.942849 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:35.944999 2193 webserver.cc:458] Webserver started at http://127.2.36.126:45553/ using document root <none> and password file <none>
I20250115 02:52:35.945407 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:35.945569 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:35.945791 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:35.946736 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "69bf2baaf0a9439f896922e654a99a3a"
format_stamp: "Formatted at 2025-01-15 02:52:35 on dist-test-slave-1lv2"
I20250115 02:52:35.950525 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20250115 02:52:35.953384 5826 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:35.954030 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:52:35.954274 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "69bf2baaf0a9439f896922e654a99a3a"
format_stamp: "Formatted at 2025-01-15 02:52:35 on dist-test-slave-1lv2"
I20250115 02:52:35.954514 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:35.973274 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:35.974184 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:36.006947 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:39537
I20250115 02:52:36.007048 5877 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:39537 every 8 connection(s)
I20250115 02:52:36.010408 5878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:36.010568 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:36.016351 5880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:36.017426 5881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:36.019330 5883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:36.019371 5878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:36.020860 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:36.024029 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:36.024225 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:36.024402 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909556024384 us; error 0 us; skew 500 ppm
I20250115 02:52:36.025079 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:36.027614 2193 webserver.cc:458] Webserver started at http://127.2.36.125:38563/ using document root <none> and password file <none>
I20250115 02:52:36.028218 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:36.028446 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:36.028751 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:36.030016 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19"
format_stamp: "Formatted at 2025-01-15 02:52:36 on dist-test-slave-1lv2"
I20250115 02:52:36.032397 5878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:36.034644 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
W20250115 02:52:36.037340 5878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:40647: Network error: Client connection negotiation failed: client connection to 127.2.36.125:40647: connect: Connection refused (error 111)
I20250115 02:52:36.038586 5891 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:36.039350 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:52:36.039613 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19"
format_stamp: "Formatted at 2025-01-15 02:52:36 on dist-test-slave-1lv2"
I20250115 02:52:36.039865 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:36.050577 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:36.051545 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:36.054824 5878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } attempt: 1
W20250115 02:52:36.059149 5878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:40647: Network error: Client connection negotiation failed: client connection to 127.2.36.125:40647: connect: Connection refused (error 111)
I20250115 02:52:36.085132 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:40647
I20250115 02:52:36.085228 5942 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:40647 every 8 connection(s)
I20250115 02:52:36.088863 5943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:36.089126 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:36.095331 5945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:36.095626 5943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:36.096266 5946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:36.096549 5878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } attempt: 2
W20250115 02:52:36.097086 5948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:36.099834 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:36.101830 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:36.102118 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:36.102353 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909556102333 us; error 0 us; skew 500 ppm
I20250115 02:52:36.102922 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:36.106513 2193 webserver.cc:458] Webserver started at http://127.2.36.124:35603/ using document root <none> and password file <none>
I20250115 02:52:36.107100 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:36.107295 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:36.107548 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:36.109010 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "403273e9fdea49c3ba74e2340193a59a"
format_stamp: "Formatted at 2025-01-15 02:52:36 on dist-test-slave-1lv2"
I20250115 02:52:36.110085 5943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:36.110364 5878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:36.114738 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
W20250115 02:52:36.115204 5878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:34495: Network error: Client connection negotiation failed: client connection to 127.2.36.124:34495: connect: Connection refused (error 111)
I20250115 02:52:36.118294 5955 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:36.119182 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:52:36.119524 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "403273e9fdea49c3ba74e2340193a59a"
format_stamp: "Formatted at 2025-01-15 02:52:36 on dist-test-slave-1lv2"
I20250115 02:52:36.119872 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:36.121083 5943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:36.132692 5878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } attempt: 1
W20250115 02:52:36.133713 5943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:34495: Network error: Client connection negotiation failed: client connection to 127.2.36.124:34495: connect: Connection refused (error 111)
W20250115 02:52:36.136628 5878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:34495: Network error: Client connection negotiation failed: client connection to 127.2.36.124:34495: connect: Connection refused (error 111)
I20250115 02:52:36.143201 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:36.144236 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:36.169052 5878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } attempt: 2
W20250115 02:52:36.173781 5878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:34495: Network error: Client connection negotiation failed: client connection to 127.2.36.124:34495: connect: Connection refused (error 111)
I20250115 02:52:36.180408 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:34495
I20250115 02:52:36.180490 6007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:34495 every 8 connection(s)
I20250115 02:52:36.209582 5943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } attempt: 1
I20250115 02:52:36.215598 6009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:36.223219 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:52:36.227406 6009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:36.236097 5943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19: Bootstrap starting.
I20250115 02:52:36.240823 6009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:36.243104 5943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:36.248451 5943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19: No bootstrap required, opened a new log
I20250115 02:52:36.251282 5943 raft_consensus.cc:357] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } }
I20250115 02:52:36.251633 6009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:36.251960 5943 raft_consensus.cc:383] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:36.252244 5943 raft_consensus.cc:738] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea7c4e8ca85d4dcd8a06bc5f801dce19, State: Initialized, Role: FOLLOWER
I20250115 02:52:36.252833 5943 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } }
I20250115 02:52:36.255152 6014 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } } }
I20250115 02:52:36.256213 6014 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:36.257061 5943 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:36.266682 6009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a: Bootstrap starting.
I20250115 02:52:36.271780 6009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a: Neither blocks nor log segments found. Creating new log.
W20250115 02:52:36.272951 6025 catalog_manager.cc:1559] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:36.273439 6025 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:36.275946 6009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a: No bootstrap required, opened a new log
I20250115 02:52:36.276409 5878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } attempt: 3
I20250115 02:52:36.278492 6009 raft_consensus.cc:357] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } }
I20250115 02:52:36.279292 6009 raft_consensus.cc:383] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:36.279577 6009 raft_consensus.cc:738] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403273e9fdea49c3ba74e2340193a59a, State: Initialized, Role: FOLLOWER
I20250115 02:52:36.280179 6009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } }
I20250115 02:52:36.282250 6027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } } }
I20250115 02:52:36.283046 6027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:36.283358 6009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:36.294350 5878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a: Bootstrap starting.
I20250115 02:52:36.296036 6014 raft_consensus.cc:491] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:36.296444 6014 raft_consensus.cc:513] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } }
I20250115 02:52:36.298295 6014 leader_election.cc:290] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69bf2baaf0a9439f896922e654a99a3a (127.2.36.126:39537), 403273e9fdea49c3ba74e2340193a59a (127.2.36.124:34495)
I20250115 02:52:36.299875 5983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403273e9fdea49c3ba74e2340193a59a" is_pre_election: true
W20250115 02:52:36.300452 6038 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250115 02:52:36.300601 5983 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea7c4e8ca85d4dcd8a06bc5f801dce19 in term 0.
W20250115 02:52:36.300822 6038 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:36.301213 5878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:36.301836 5893 leader_election.cc:304] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403273e9fdea49c3ba74e2340193a59a, ea7c4e8ca85d4dcd8a06bc5f801dce19; no voters:
I20250115 02:52:36.302450 6014 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:36.302709 6014 raft_consensus.cc:491] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:36.302980 6014 raft_consensus.cc:3054] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:36.306022 5878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a: No bootstrap required, opened a new log
I20250115 02:52:36.308254 5878 raft_consensus.cc:357] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } }
I20250115 02:52:36.308763 5878 raft_consensus.cc:383] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:36.308988 5878 raft_consensus.cc:738] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69bf2baaf0a9439f896922e654a99a3a, State: Initialized, Role: FOLLOWER
I20250115 02:52:36.309530 5878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } }
I20250115 02:52:36.310777 6014 raft_consensus.cc:513] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } }
I20250115 02:52:36.311503 6040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } } }
I20250115 02:52:36.312271 6040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:36.313098 6014 leader_election.cc:290] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [CANDIDATE]: Term 1 election: Requested vote from peers 69bf2baaf0a9439f896922e654a99a3a (127.2.36.126:39537), 403273e9fdea49c3ba74e2340193a59a (127.2.36.124:34495)
I20250115 02:52:36.313238 5878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:36.313866 5852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69bf2baaf0a9439f896922e654a99a3a"
I20250115 02:52:36.314090 5983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403273e9fdea49c3ba74e2340193a59a"
I20250115 02:52:36.314492 5852 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:36.314590 5983 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:36.322116 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69bf2baaf0a9439f896922e654a99a3a" is_pre_election: true
I20250115 02:52:36.322690 5983 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea7c4e8ca85d4dcd8a06bc5f801dce19 in term 1.
I20250115 02:52:36.324338 5893 leader_election.cc:304] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403273e9fdea49c3ba74e2340193a59a, ea7c4e8ca85d4dcd8a06bc5f801dce19; no voters:
I20250115 02:52:36.324972 6014 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:36.326088 6014 raft_consensus.cc:695] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 1 LEADER]: Becoming Leader. State: Replica: ea7c4e8ca85d4dcd8a06bc5f801dce19, State: Running, Role: LEADER
I20250115 02:52:36.326261 5852 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea7c4e8ca85d4dcd8a06bc5f801dce19 in term 1.
I20250115 02:52:36.327107 6014 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } }
I20250115 02:52:36.331518 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:52:36.331813 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:52:36.333583 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:36.333660 6051 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea7c4e8ca85d4dcd8a06bc5f801dce19. Latest consensus state: current_term: 1 leader_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } } }
I20250115 02:52:36.334542 6051 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [sys.catalog]: This master's current role is: LEADER
W20250115 02:52:36.336529 6053 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:36.336916 6053 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:36.336958 6054 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
W20250115 02:52:36.342784 6055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:36.343659 6056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:36.345490 6054 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250115 02:52:36.350714 6058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:36.352427 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:36.353289 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:36.353541 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:36.353745 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909556353727 us; error 0 us; skew 500 ppm
I20250115 02:52:36.354425 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:36.355646 5983 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [term 1 FOLLOWER]: Refusing update from remote peer ea7c4e8ca85d4dcd8a06bc5f801dce19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:36.356415 5852 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [term 1 FOLLOWER]: Refusing update from remote peer ea7c4e8ca85d4dcd8a06bc5f801dce19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:36.357227 6014 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:36.357975 6051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:36.358487 2193 webserver.cc:458] Webserver started at http://127.2.36.65:44893/ using document root <none> and password file <none>
I20250115 02:52:36.359210 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:36.359479 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:36.359834 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:36.361321 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "324993df23604663b8962b32ebf424fc"
format_stamp: "Formatted at 2025-01-15 02:52:36 on dist-test-slave-1lv2"
I20250115 02:52:36.370409 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.008s
I20250115 02:52:36.373560 6027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea7c4e8ca85d4dcd8a06bc5f801dce19. Latest consensus state: current_term: 1 leader_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } } }
I20250115 02:52:36.374403 6027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:36.375672 6068 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:36.377552 6040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea7c4e8ca85d4dcd8a06bc5f801dce19. Latest consensus state: current_term: 1 leader_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } } }
I20250115 02:52:36.378330 6040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:36.378144 6051 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } } }
I20250115 02:52:36.379114 6051 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:36.380371 2193 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20250115 02:52:36.380438 6054 catalog_manager.cc:1348] Generated new cluster ID: 0550ec1161584719b1f927657aba841a
I20250115 02:52:36.380995 6054 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:52:36.381037 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "324993df23604663b8962b32ebf424fc"
format_stamp: "Formatted at 2025-01-15 02:52:36 on dist-test-slave-1lv2"
I20250115 02:52:36.381460 6051 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } } }
I20250115 02:52:36.381755 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:36.382247 6051 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:36.387113 6027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } } }
I20250115 02:52:36.387840 6027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:36.396173 6040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf2baaf0a9439f896922e654a99a3a" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39537 } } peers { permanent_uuid: "ea7c4e8ca85d4dcd8a06bc5f801dce19" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40647 } } peers { permanent_uuid: "403273e9fdea49c3ba74e2340193a59a" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 34495 } } }
I20250115 02:52:36.396914 6040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:36.413125 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:36.414031 6054 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:52:36.414950 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:36.415908 6054 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:52:36.423470 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:36.426604 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:36.426895 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250115 02:52:36.427191 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:36.427407 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:36.439735 6054 catalog_manager.cc:5899] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19: Generated new TSK 0
I20250115 02:52:36.440550 6054 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:52:36.476297 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:41111
I20250115 02:52:36.476387 6134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:41111 every 8 connection(s)
I20250115 02:52:36.496096 6135 heartbeater.cc:346] Connected to a master server at 127.2.36.124:34495
I20250115 02:52:36.496582 6135 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:36.497573 6135 heartbeater.cc:509] Master 127.2.36.124:34495 requested a full tablet report, sending...
I20250115 02:52:36.500805 5973 ts_manager.cc:194] Registered new tserver with Master: 324993df23604663b8962b32ebf424fc (127.2.36.65:41111)
I20250115 02:52:36.506300 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:36.515857 6137 heartbeater.cc:346] Connected to a master server at 127.2.36.126:39537
I20250115 02:52:36.516439 6137 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:36.517269 6137 heartbeater.cc:509] Master 127.2.36.126:39537 requested a full tablet report, sending...
I20250115 02:52:36.515998 6136 heartbeater.cc:346] Connected to a master server at 127.2.36.125:40647
I20250115 02:52:36.518218 6136 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:36.519223 6136 heartbeater.cc:509] Master 127.2.36.125:40647 requested a full tablet report, sending...
W20250115 02:52:36.519996 6142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:36.520257 6143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:36.522464 5843 ts_manager.cc:194] Registered new tserver with Master: 324993df23604663b8962b32ebf424fc (127.2.36.65:41111)
I20250115 02:52:36.522411 5908 ts_manager.cc:194] Registered new tserver with Master: 324993df23604663b8962b32ebf424fc (127.2.36.65:41111)
W20250115 02:52:36.524189 6145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:36.524822 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:36.525223 5908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36246
I20250115 02:52:36.525701 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:36.525934 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:36.526142 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909556526122 us; error 0 us; skew 500 ppm
I20250115 02:52:36.526932 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:36.529860 2193 webserver.cc:458] Webserver started at http://127.2.36.66:40345/ using document root <none> and password file <none>
I20250115 02:52:36.530315 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:36.530491 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:36.530731 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:36.531924 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "acca59c4e7cb4a1c8bced20605d3d75c"
format_stamp: "Formatted at 2025-01-15 02:52:36 on dist-test-slave-1lv2"
I20250115 02:52:36.536599 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:52:36.537972 6136 heartbeater.cc:501] Master 127.2.36.125:40647 was elected leader, sending a full tablet report...
I20250115 02:52:36.540145 6150 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:36.541302 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250115 02:52:36.541649 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "acca59c4e7cb4a1c8bced20605d3d75c"
format_stamp: "Formatted at 2025-01-15 02:52:36 on dist-test-slave-1lv2"
I20250115 02:52:36.542008 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:36.560864 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:36.562049 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:36.563446 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:36.566197 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:36.566453 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:36.566730 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:36.566963 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20250115 02:52:36.612558 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:34769
I20250115 02:52:36.612787 6212 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:34769 every 8 connection(s)
I20250115 02:52:36.635747 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:36.650499 6213 heartbeater.cc:346] Connected to a master server at 127.2.36.124:34495
I20250115 02:52:36.654280 6213 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:36.655202 6213 heartbeater.cc:509] Master 127.2.36.124:34495 requested a full tablet report, sending...
I20250115 02:52:36.653394 6214 heartbeater.cc:346] Connected to a master server at 127.2.36.125:40647
I20250115 02:52:36.656134 6214 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:36.657002 6214 heartbeater.cc:509] Master 127.2.36.125:40647 requested a full tablet report, sending...
I20250115 02:52:36.657856 5973 ts_manager.cc:194] Registered new tserver with Master: acca59c4e7cb4a1c8bced20605d3d75c (127.2.36.66:34769)
I20250115 02:52:36.659488 5907 ts_manager.cc:194] Registered new tserver with Master: acca59c4e7cb4a1c8bced20605d3d75c (127.2.36.66:34769)
I20250115 02:52:36.661540 5907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36254
W20250115 02:52:36.665227 6220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:36.665499 6215 heartbeater.cc:346] Connected to a master server at 127.2.36.126:39537
I20250115 02:52:36.669003 6215 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:36.669670 6215 heartbeater.cc:509] Master 127.2.36.126:39537 requested a full tablet report, sending...
I20250115 02:52:36.674425 6214 heartbeater.cc:501] Master 127.2.36.125:40647 was elected leader, sending a full tablet report...
I20250115 02:52:36.674798 5843 ts_manager.cc:194] Registered new tserver with Master: acca59c4e7cb4a1c8bced20605d3d75c (127.2.36.66:34769)
W20250115 02:52:36.676450 6221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:36.681438 6223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:36.684398 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:36.685470 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:36.685739 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:36.685956 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909556685938 us; error 0 us; skew 500 ppm
I20250115 02:52:36.687152 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:36.689640 2193 webserver.cc:458] Webserver started at http://127.2.36.67:35077/ using document root <none> and password file <none>
I20250115 02:52:36.690147 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:36.690338 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:36.690603 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:36.691939 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "33d4224a68324476b984325a0db0e8a3"
format_stamp: "Formatted at 2025-01-15 02:52:36 on dist-test-slave-1lv2"
I20250115 02:52:36.696620 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:52:36.700605 6228 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:36.701498 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250115 02:52:36.701792 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "33d4224a68324476b984325a0db0e8a3"
format_stamp: "Formatted at 2025-01-15 02:52:36 on dist-test-slave-1lv2"
I20250115 02:52:36.702060 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshot.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:36.728734 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:36.730209 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:36.731958 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:36.734660 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:36.734927 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250115 02:52:36.735204 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:36.735404 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:36.806473 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:40389
I20250115 02:52:36.806543 6290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:40389 every 8 connection(s)
I20250115 02:52:36.832177 6291 heartbeater.cc:346] Connected to a master server at 127.2.36.124:34495
I20250115 02:52:36.832518 6291 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:36.833292 6291 heartbeater.cc:509] Master 127.2.36.124:34495 requested a full tablet report, sending...
I20250115 02:52:36.835502 6292 heartbeater.cc:346] Connected to a master server at 127.2.36.125:40647
I20250115 02:52:36.835912 6292 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:36.836985 6292 heartbeater.cc:509] Master 127.2.36.125:40647 requested a full tablet report, sending...
I20250115 02:52:36.835901 5972 ts_manager.cc:194] Registered new tserver with Master: 33d4224a68324476b984325a0db0e8a3 (127.2.36.67:40389)
I20250115 02:52:36.839480 5908 ts_manager.cc:194] Registered new tserver with Master: 33d4224a68324476b984325a0db0e8a3 (127.2.36.67:40389)
I20250115 02:52:36.840864 5908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36262
I20250115 02:52:36.848594 6294 heartbeater.cc:346] Connected to a master server at 127.2.36.126:39537
I20250115 02:52:36.848933 6294 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:36.849539 6294 heartbeater.cc:509] Master 127.2.36.126:39537 requested a full tablet report, sending...
I20250115 02:52:36.851931 5843 ts_manager.cc:194] Registered new tserver with Master: 33d4224a68324476b984325a0db0e8a3 (127.2.36.67:40389)
I20250115 02:52:36.852589 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.024896843s
I20250115 02:52:36.853534 6292 heartbeater.cc:501] Master 127.2.36.125:40647 was elected leader, sending a full tablet report...
I20250115 02:52:36.903360 5907 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36276:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:52:36.906121 5907 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:52:37.006667 6178 tablet_service.cc:1467] Processing CreateTablet for tablet d92e45d9f4e34b46ad5b9d3feea875bd (DEFAULT_TABLE table=ksck-test-table [id=0e57cb33449a4916832e4c3ef1285750]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:37.008423 6177 tablet_service.cc:1467] Processing CreateTablet for tablet 5360b5e930e34cbbb0a84f0b662324bd (DEFAULT_TABLE table=ksck-test-table [id=0e57cb33449a4916832e4c3ef1285750]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:37.009738 6176 tablet_service.cc:1467] Processing CreateTablet for tablet a1ac97a302e849ae8b6ac6d8214f347f (DEFAULT_TABLE table=ksck-test-table [id=0e57cb33449a4916832e4c3ef1285750]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:37.016419 6178 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92e45d9f4e34b46ad5b9d3feea875bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:37.018467 6176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ac97a302e849ae8b6ac6d8214f347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:37.019073 6256 tablet_service.cc:1467] Processing CreateTablet for tablet d92e45d9f4e34b46ad5b9d3feea875bd (DEFAULT_TABLE table=ksck-test-table [id=0e57cb33449a4916832e4c3ef1285750]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:37.019690 6254 tablet_service.cc:1467] Processing CreateTablet for tablet a1ac97a302e849ae8b6ac6d8214f347f (DEFAULT_TABLE table=ksck-test-table [id=0e57cb33449a4916832e4c3ef1285750]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:37.020291 6256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92e45d9f4e34b46ad5b9d3feea875bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:37.020933 6254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ac97a302e849ae8b6ac6d8214f347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:37.022136 6255 tablet_service.cc:1467] Processing CreateTablet for tablet 5360b5e930e34cbbb0a84f0b662324bd (DEFAULT_TABLE table=ksck-test-table [id=0e57cb33449a4916832e4c3ef1285750]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:37.023265 6255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5360b5e930e34cbbb0a84f0b662324bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:37.026677 6100 tablet_service.cc:1467] Processing CreateTablet for tablet d92e45d9f4e34b46ad5b9d3feea875bd (DEFAULT_TABLE table=ksck-test-table [id=0e57cb33449a4916832e4c3ef1285750]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:37.027948 6100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92e45d9f4e34b46ad5b9d3feea875bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:37.028391 6177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5360b5e930e34cbbb0a84f0b662324bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:37.038009 6099 tablet_service.cc:1467] Processing CreateTablet for tablet 5360b5e930e34cbbb0a84f0b662324bd (DEFAULT_TABLE table=ksck-test-table [id=0e57cb33449a4916832e4c3ef1285750]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:37.039583 6099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5360b5e930e34cbbb0a84f0b662324bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:37.039284 6098 tablet_service.cc:1467] Processing CreateTablet for tablet a1ac97a302e849ae8b6ac6d8214f347f (DEFAULT_TABLE table=ksck-test-table [id=0e57cb33449a4916832e4c3ef1285750]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:37.041821 6098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ac97a302e849ae8b6ac6d8214f347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:37.069619 6313 tablet_bootstrap.cc:492] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c: Bootstrap starting.
I20250115 02:52:37.082806 6313 tablet_bootstrap.cc:654] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:37.098138 6314 tablet_bootstrap.cc:492] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3: Bootstrap starting.
I20250115 02:52:37.117295 6316 tablet_bootstrap.cc:492] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc: Bootstrap starting.
I20250115 02:52:37.125259 6313 tablet_bootstrap.cc:492] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c: No bootstrap required, opened a new log
I20250115 02:52:37.125695 6313 ts_tablet_manager.cc:1397] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.009s
I20250115 02:52:37.127874 6313 raft_consensus.cc:357] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.128396 6313 raft_consensus.cc:383] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:37.128647 6313 raft_consensus.cc:738] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acca59c4e7cb4a1c8bced20605d3d75c, State: Initialized, Role: FOLLOWER
I20250115 02:52:37.128753 6316 tablet_bootstrap.cc:654] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:37.129513 6314 tablet_bootstrap.cc:654] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:37.132320 6313 consensus_queue.cc:260] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.150100 6314 tablet_bootstrap.cc:492] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3: No bootstrap required, opened a new log
I20250115 02:52:37.150625 6314 ts_tablet_manager.cc:1397] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.004s
I20250115 02:52:37.156826 6316 tablet_bootstrap.cc:492] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc: No bootstrap required, opened a new log
I20250115 02:52:37.155653 6314 raft_consensus.cc:357] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.157310 6316 ts_tablet_manager.cc:1397] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.012s
I20250115 02:52:37.158011 6313 ts_tablet_manager.cc:1428] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c: Time spent starting tablet: real 0.032s user 0.019s sys 0.005s
I20250115 02:52:37.159190 6313 tablet_bootstrap.cc:492] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c: Bootstrap starting.
I20250115 02:52:37.165369 6313 tablet_bootstrap.cc:654] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:37.165030 6316 raft_consensus.cc:357] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.165760 6316 raft_consensus.cc:383] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:37.166078 6316 raft_consensus.cc:738] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324993df23604663b8962b32ebf424fc, State: Initialized, Role: FOLLOWER
I20250115 02:52:37.166934 6316 consensus_queue.cc:260] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.168118 6314 raft_consensus.cc:383] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:37.168483 6314 raft_consensus.cc:738] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d4224a68324476b984325a0db0e8a3, State: Initialized, Role: FOLLOWER
I20250115 02:52:37.169245 6314 consensus_queue.cc:260] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.198115 6316 ts_tablet_manager.cc:1428] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc: Time spent starting tablet: real 0.040s user 0.009s sys 0.001s
I20250115 02:52:37.206112 6314 ts_tablet_manager.cc:1428] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3: Time spent starting tablet: real 0.055s user 0.010s sys 0.027s
I20250115 02:52:37.207377 6314 tablet_bootstrap.cc:492] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3: Bootstrap starting.
I20250115 02:52:37.208947 6313 tablet_bootstrap.cc:492] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c: No bootstrap required, opened a new log
I20250115 02:52:37.209431 6313 ts_tablet_manager.cc:1397] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c: Time spent bootstrapping tablet: real 0.050s user 0.019s sys 0.013s
I20250115 02:52:37.212199 6313 raft_consensus.cc:357] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.212904 6313 raft_consensus.cc:383] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:37.213213 6313 raft_consensus.cc:738] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acca59c4e7cb4a1c8bced20605d3d75c, State: Initialized, Role: FOLLOWER
I20250115 02:52:37.214033 6314 tablet_bootstrap.cc:654] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:37.214035 6313 consensus_queue.cc:260] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.222263 6313 ts_tablet_manager.cc:1428] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20250115 02:52:37.225582 6314 tablet_bootstrap.cc:492] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3: No bootstrap required, opened a new log
I20250115 02:52:37.226154 6314 ts_tablet_manager.cc:1397] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20250115 02:52:37.226819 6313 tablet_bootstrap.cc:492] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c: Bootstrap starting.
I20250115 02:52:37.229079 6314 raft_consensus.cc:357] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.229982 6314 raft_consensus.cc:383] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:37.230316 6314 raft_consensus.cc:738] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d4224a68324476b984325a0db0e8a3, State: Initialized, Role: FOLLOWER
I20250115 02:52:37.231158 6314 consensus_queue.cc:260] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.235248 6313 tablet_bootstrap.cc:654] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:37.236043 6314 ts_tablet_manager.cc:1428] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20250115 02:52:37.237107 6314 tablet_bootstrap.cc:492] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3: Bootstrap starting.
I20250115 02:52:37.243333 6314 tablet_bootstrap.cc:654] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:37.254854 6314 tablet_bootstrap.cc:492] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3: No bootstrap required, opened a new log
I20250115 02:52:37.255431 6314 ts_tablet_manager.cc:1397] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20250115 02:52:37.256790 6316 tablet_bootstrap.cc:492] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc: Bootstrap starting.
I20250115 02:52:37.258419 6314 raft_consensus.cc:357] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.259351 6314 raft_consensus.cc:383] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:37.259734 6314 raft_consensus.cc:738] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d4224a68324476b984325a0db0e8a3, State: Initialized, Role: FOLLOWER
I20250115 02:52:37.260764 6314 consensus_queue.cc:260] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.267089 6314 ts_tablet_manager.cc:1428] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250115 02:52:37.285336 6316 tablet_bootstrap.cc:654] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:37.284087 6313 tablet_bootstrap.cc:492] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c: No bootstrap required, opened a new log
I20250115 02:52:37.288610 6313 ts_tablet_manager.cc:1397] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.037s
I20250115 02:52:37.291563 6313 raft_consensus.cc:357] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.292320 6313 raft_consensus.cc:383] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:37.292619 6313 raft_consensus.cc:738] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acca59c4e7cb4a1c8bced20605d3d75c, State: Initialized, Role: FOLLOWER
I20250115 02:52:37.293362 6313 consensus_queue.cc:260] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.300318 6316 tablet_bootstrap.cc:492] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc: No bootstrap required, opened a new log
I20250115 02:52:37.300899 6316 ts_tablet_manager.cc:1397] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.006s
I20250115 02:52:37.304442 6316 raft_consensus.cc:357] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.306177 6038 catalog_manager.cc:1260] Loaded cluster ID: 0550ec1161584719b1f927657aba841a
I20250115 02:52:37.306473 6038 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a: loading cluster ID for follower catalog manager: success
I20250115 02:52:37.307221 6316 raft_consensus.cc:383] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:37.307653 6316 raft_consensus.cc:738] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324993df23604663b8962b32ebf424fc, State: Initialized, Role: FOLLOWER
I20250115 02:52:37.308346 6316 consensus_queue.cc:260] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.310364 6316 ts_tablet_manager.cc:1428] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc: Time spent starting tablet: real 0.009s user 0.002s sys 0.002s
I20250115 02:52:37.311339 6316 tablet_bootstrap.cc:492] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc: Bootstrap starting.
I20250115 02:52:37.312862 6038 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a: acquiring CA information for follower catalog manager: success
I20250115 02:52:37.315083 6313 ts_tablet_manager.cc:1428] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20250115 02:52:37.320596 6316 tablet_bootstrap.cc:654] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:37.327637 6038 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:37.340814 6316 tablet_bootstrap.cc:492] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc: No bootstrap required, opened a new log
I20250115 02:52:37.341333 6316 ts_tablet_manager.cc:1397] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.011s
I20250115 02:52:37.342226 6053 catalog_manager.cc:1260] Loaded cluster ID: 0550ec1161584719b1f927657aba841a
I20250115 02:52:37.342525 6053 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a: loading cluster ID for follower catalog manager: success
I20250115 02:52:37.349082 6316 raft_consensus.cc:357] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.349843 6316 raft_consensus.cc:383] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:37.350127 6316 raft_consensus.cc:738] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324993df23604663b8962b32ebf424fc, State: Initialized, Role: FOLLOWER
I20250115 02:52:37.350872 6316 consensus_queue.cc:260] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.353220 6316 ts_tablet_manager.cc:1428] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250115 02:52:37.354681 6053 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a: acquiring CA information for follower catalog manager: success
I20250115 02:52:37.365792 6053 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:37.378846 6320 raft_consensus.cc:491] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:37.379366 6320 raft_consensus.cc:513] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.381556 6320 leader_election.cc:290] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acca59c4e7cb4a1c8bced20605d3d75c (127.2.36.66:34769), 33d4224a68324476b984325a0db0e8a3 (127.2.36.67:40389)
I20250115 02:52:37.383240 6319 raft_consensus.cc:491] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:37.383716 6319 raft_consensus.cc:513] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.386073 6319 leader_election.cc:290] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 324993df23604663b8962b32ebf424fc (127.2.36.65:41111), 33d4224a68324476b984325a0db0e8a3 (127.2.36.67:40389)
I20250115 02:52:37.393162 6319 raft_consensus.cc:491] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:37.393599 6319 raft_consensus.cc:513] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.395459 6319 leader_election.cc:290] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 324993df23604663b8962b32ebf424fc (127.2.36.65:41111), 33d4224a68324476b984325a0db0e8a3 (127.2.36.67:40389)
I20250115 02:52:37.405561 6188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92e45d9f4e34b46ad5b9d3feea875bd" candidate_uuid: "324993df23604663b8962b32ebf424fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" is_pre_election: true
I20250115 02:52:37.406378 6188 raft_consensus.cc:2463] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324993df23604663b8962b32ebf424fc in term 0.
I20250115 02:52:37.407773 6074 leader_election.cc:304] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324993df23604663b8962b32ebf424fc, acca59c4e7cb4a1c8bced20605d3d75c; no voters:
I20250115 02:52:37.408675 6320 raft_consensus.cc:2798] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:37.409056 6320 raft_consensus.cc:491] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:37.409397 6320 raft_consensus.cc:3054] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:37.423100 6320 raft_consensus.cc:513] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.429253 6320 leader_election.cc:290] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [CANDIDATE]: Term 1 election: Requested vote from peers acca59c4e7cb4a1c8bced20605d3d75c (127.2.36.66:34769), 33d4224a68324476b984325a0db0e8a3 (127.2.36.67:40389)
I20250115 02:52:37.431263 6188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92e45d9f4e34b46ad5b9d3feea875bd" candidate_uuid: "324993df23604663b8962b32ebf424fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acca59c4e7cb4a1c8bced20605d3d75c"
I20250115 02:52:37.431885 6188 raft_consensus.cc:3054] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:37.437208 6320 raft_consensus.cc:491] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:37.437790 6320 raft_consensus.cc:513] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.439476 6110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ac97a302e849ae8b6ac6d8214f347f" candidate_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324993df23604663b8962b32ebf424fc" is_pre_election: true
I20250115 02:52:37.440222 6110 raft_consensus.cc:2463] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acca59c4e7cb4a1c8bced20605d3d75c in term 0.
I20250115 02:52:37.440900 6266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92e45d9f4e34b46ad5b9d3feea875bd" candidate_uuid: "324993df23604663b8962b32ebf424fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d4224a68324476b984325a0db0e8a3" is_pre_election: true
I20250115 02:52:37.441562 6266 raft_consensus.cc:2463] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324993df23604663b8962b32ebf424fc in term 0.
I20250115 02:52:37.441602 6265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92e45d9f4e34b46ad5b9d3feea875bd" candidate_uuid: "324993df23604663b8962b32ebf424fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d4224a68324476b984325a0db0e8a3"
I20250115 02:52:37.441953 6110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92e45d9f4e34b46ad5b9d3feea875bd" candidate_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324993df23604663b8962b32ebf424fc" is_pre_election: true
I20250115 02:52:37.442669 6320 leader_election.cc:290] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acca59c4e7cb4a1c8bced20605d3d75c (127.2.36.66:34769), 33d4224a68324476b984325a0db0e8a3 (127.2.36.67:40389)
I20250115 02:52:37.442780 6110 raft_consensus.cc:2388] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acca59c4e7cb4a1c8bced20605d3d75c in current term 1: Already voted for candidate 324993df23604663b8962b32ebf424fc in this term.
I20250115 02:52:37.443202 6264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ac97a302e849ae8b6ac6d8214f347f" candidate_uuid: "324993df23604663b8962b32ebf424fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d4224a68324476b984325a0db0e8a3" is_pre_election: true
I20250115 02:52:37.438694 6188 raft_consensus.cc:2463] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 324993df23604663b8962b32ebf424fc in term 1.
I20250115 02:52:37.444717 6074 leader_election.cc:304] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324993df23604663b8962b32ebf424fc, acca59c4e7cb4a1c8bced20605d3d75c; no voters:
I20250115 02:52:37.446297 6264 raft_consensus.cc:2463] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324993df23604663b8962b32ebf424fc in term 0.
I20250115 02:52:37.447075 6320 raft_consensus.cc:2798] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:37.447937 6152 leader_election.cc:304] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324993df23604663b8962b32ebf424fc, acca59c4e7cb4a1c8bced20605d3d75c; no voters:
I20250115 02:52:37.448712 6319 raft_consensus.cc:2798] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:37.449056 6319 raft_consensus.cc:491] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:37.449371 6319 raft_consensus.cc:3054] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:37.450158 6265 raft_consensus.cc:3054] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:37.451494 6322 raft_consensus.cc:491] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:37.451946 6322 raft_consensus.cc:513] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.454304 6322 leader_election.cc:290] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 324993df23604663b8962b32ebf424fc (127.2.36.65:41111), acca59c4e7cb4a1c8bced20605d3d75c (127.2.36.66:34769)
I20250115 02:52:37.442598 6187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ac97a302e849ae8b6ac6d8214f347f" candidate_uuid: "324993df23604663b8962b32ebf424fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" is_pre_election: true
I20250115 02:52:37.461230 6320 raft_consensus.cc:695] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 1 LEADER]: Becoming Leader. State: Replica: 324993df23604663b8962b32ebf424fc, State: Running, Role: LEADER
I20250115 02:52:37.464097 6073 leader_election.cc:304] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324993df23604663b8962b32ebf424fc, 33d4224a68324476b984325a0db0e8a3; no voters:
I20250115 02:52:37.462152 6320 consensus_queue.cc:237] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.469388 6319 raft_consensus.cc:513] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.470536 6187 raft_consensus.cc:2388] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 324993df23604663b8962b32ebf424fc in current term 1: Already voted for candidate acca59c4e7cb4a1c8bced20605d3d75c in this term.
I20250115 02:52:37.474464 6265 raft_consensus.cc:2463] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 324993df23604663b8962b32ebf424fc in term 1.
I20250115 02:52:37.483186 6319 leader_election.cc:290] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [CANDIDATE]: Term 1 election: Requested vote from peers 324993df23604663b8962b32ebf424fc (127.2.36.65:41111), 33d4224a68324476b984325a0db0e8a3 (127.2.36.67:40389)
I20250115 02:52:37.484741 6333 raft_consensus.cc:2798] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:37.485172 6333 raft_consensus.cc:491] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:37.485498 6333 raft_consensus.cc:3054] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:37.488659 6110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ac97a302e849ae8b6ac6d8214f347f" candidate_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324993df23604663b8962b32ebf424fc"
I20250115 02:52:37.491139 6264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92e45d9f4e34b46ad5b9d3feea875bd" candidate_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d4224a68324476b984325a0db0e8a3" is_pre_election: true
I20250115 02:52:37.492285 6264 raft_consensus.cc:2388] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acca59c4e7cb4a1c8bced20605d3d75c in current term 1: Already voted for candidate 324993df23604663b8962b32ebf424fc in this term.
I20250115 02:52:37.495105 6264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ac97a302e849ae8b6ac6d8214f347f" candidate_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d4224a68324476b984325a0db0e8a3"
I20250115 02:52:37.495609 6333 raft_consensus.cc:513] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.496886 6110 raft_consensus.cc:2388] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate acca59c4e7cb4a1c8bced20605d3d75c in current term 1: Already voted for candidate 324993df23604663b8962b32ebf424fc in this term.
I20250115 02:52:37.495667 6264 raft_consensus.cc:3054] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:37.498940 6187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ac97a302e849ae8b6ac6d8214f347f" candidate_uuid: "324993df23604663b8962b32ebf424fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acca59c4e7cb4a1c8bced20605d3d75c"
I20250115 02:52:37.496568 6152 leader_election.cc:304] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acca59c4e7cb4a1c8bced20605d3d75c; no voters: 324993df23604663b8962b32ebf424fc, 33d4224a68324476b984325a0db0e8a3
I20250115 02:52:37.502539 6187 raft_consensus.cc:2388] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 324993df23604663b8962b32ebf424fc in current term 1: Already voted for candidate acca59c4e7cb4a1c8bced20605d3d75c in this term.
I20250115 02:52:37.502810 6266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ac97a302e849ae8b6ac6d8214f347f" candidate_uuid: "324993df23604663b8962b32ebf424fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d4224a68324476b984325a0db0e8a3"
I20250115 02:52:37.503278 6319 raft_consensus.cc:2743] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:52:37.504254 6073 leader_election.cc:304] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 324993df23604663b8962b32ebf424fc; no voters: 33d4224a68324476b984325a0db0e8a3, acca59c4e7cb4a1c8bced20605d3d75c
I20250115 02:52:37.505640 6110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360b5e930e34cbbb0a84f0b662324bd" candidate_uuid: "33d4224a68324476b984325a0db0e8a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324993df23604663b8962b32ebf424fc" is_pre_election: true
I20250115 02:52:37.506265 6110 raft_consensus.cc:2463] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d4224a68324476b984325a0db0e8a3 in term 0.
I20250115 02:52:37.507735 6264 raft_consensus.cc:2463] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acca59c4e7cb4a1c8bced20605d3d75c in term 1.
I20250115 02:52:37.491428 6265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ac97a302e849ae8b6ac6d8214f347f" candidate_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d4224a68324476b984325a0db0e8a3" is_pre_election: true
I20250115 02:52:37.508812 6320 raft_consensus.cc:2743] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250115 02:52:37.501848 6333 leader_election.cc:290] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [CANDIDATE]: Term 1 election: Requested vote from peers acca59c4e7cb4a1c8bced20605d3d75c (127.2.36.66:34769), 33d4224a68324476b984325a0db0e8a3 (127.2.36.67:40389)
I20250115 02:52:37.489951 5906 catalog_manager.cc:5526] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 324993df23604663b8962b32ebf424fc (127.2.36.65). New cstate: current_term: 1 leader_uuid: "324993df23604663b8962b32ebf424fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:37.512725 6152 leader_election.cc:304] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33d4224a68324476b984325a0db0e8a3, acca59c4e7cb4a1c8bced20605d3d75c; no voters: 324993df23604663b8962b32ebf424fc
I20250115 02:52:37.513883 6319 raft_consensus.cc:2798] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:37.515184 6230 leader_election.cc:304] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324993df23604663b8962b32ebf424fc, 33d4224a68324476b984325a0db0e8a3; no voters:
I20250115 02:52:37.508836 6265 raft_consensus.cc:2371] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate acca59c4e7cb4a1c8bced20605d3d75c in term 1. Re-sending same reply.
I20250115 02:52:37.517783 6322 raft_consensus.cc:2798] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:37.518160 6322 raft_consensus.cc:491] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:37.518479 6322 raft_consensus.cc:3054] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:37.525758 6322 raft_consensus.cc:513] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.528115 6319 raft_consensus.cc:695] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 1 LEADER]: Becoming Leader. State: Replica: acca59c4e7cb4a1c8bced20605d3d75c, State: Running, Role: LEADER
I20250115 02:52:37.528102 6187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360b5e930e34cbbb0a84f0b662324bd" candidate_uuid: "33d4224a68324476b984325a0db0e8a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" is_pre_election: true
I20250115 02:52:37.529057 6187 raft_consensus.cc:2463] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d4224a68324476b984325a0db0e8a3 in term 0.
I20250115 02:52:37.531477 6322 leader_election.cc:290] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [CANDIDATE]: Term 1 election: Requested vote from peers 324993df23604663b8962b32ebf424fc (127.2.36.65:41111), acca59c4e7cb4a1c8bced20605d3d75c (127.2.36.66:34769)
I20250115 02:52:37.532940 6110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360b5e930e34cbbb0a84f0b662324bd" candidate_uuid: "33d4224a68324476b984325a0db0e8a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324993df23604663b8962b32ebf424fc"
I20250115 02:52:37.533545 6110 raft_consensus.cc:3054] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:37.533985 6187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5360b5e930e34cbbb0a84f0b662324bd" candidate_uuid: "33d4224a68324476b984325a0db0e8a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acca59c4e7cb4a1c8bced20605d3d75c"
I20250115 02:52:37.534571 6187 raft_consensus.cc:3054] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:37.529278 6319 consensus_queue.cc:237] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.540676 6110 raft_consensus.cc:2463] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d4224a68324476b984325a0db0e8a3 in term 1.
I20250115 02:52:37.541381 6187 raft_consensus.cc:2463] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d4224a68324476b984325a0db0e8a3 in term 1.
I20250115 02:52:37.541973 6230 leader_election.cc:304] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324993df23604663b8962b32ebf424fc, 33d4224a68324476b984325a0db0e8a3; no voters:
I20250115 02:52:37.543298 6322 raft_consensus.cc:2798] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:37.552659 6322 raft_consensus.cc:695] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 1 LEADER]: Becoming Leader. State: Replica: 33d4224a68324476b984325a0db0e8a3, State: Running, Role: LEADER
I20250115 02:52:37.553507 6322 consensus_queue.cc:237] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } }
I20250115 02:52:37.561080 5907 catalog_manager.cc:5526] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c reported cstate change: term changed from 0 to 1, leader changed from <none> to acca59c4e7cb4a1c8bced20605d3d75c (127.2.36.66). New cstate: current_term: 1 leader_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:37.606845 5905 catalog_manager.cc:5526] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33d4224a68324476b984325a0db0e8a3 (127.2.36.67). New cstate: current_term: 1 leader_uuid: "33d4224a68324476b984325a0db0e8a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 } health_report { overall_health: HEALTHY } } }
I20250115 02:52:37.799854 6265 raft_consensus.cc:1270] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3 [term 1 FOLLOWER]: Refusing update from remote peer 324993df23604663b8962b32ebf424fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:37.801376 6187 raft_consensus.cc:1270] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Refusing update from remote peer 324993df23604663b8962b32ebf424fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:37.801345 6333 consensus_queue.cc:1035] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250115 02:52:37.806592 6320 consensus_queue.cc:1035] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:37.990006 6322 consensus_queue.cc:1035] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acca59c4e7cb4a1c8bced20605d3d75c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:38.008554 6322 consensus_queue.cc:1035] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:38.114578 6319 consensus_queue.cc:1035] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d4224a68324476b984325a0db0e8a3" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:38.150885 6319 consensus_queue.cc:1035] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "324993df23604663b8962b32ebf424fc" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:41.572803 6176 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:52:41.592612 6100 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:52:41.607693 6254 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20250115 02:52:42.437563 6350 ksck_checksum.cc:286] failed to submit task to start additional tablet checksums: Service unavailable: The pool has been shut down.
I20250115 02:52:42.463425 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:52:42.509007 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:42.509642 2193 tablet_replica.cc:331] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc: stopping tablet replica
I20250115 02:52:42.511163 2193 raft_consensus.cc:2238] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:42.511668 2193 raft_consensus.cc:2267] T 5360b5e930e34cbbb0a84f0b662324bd P 324993df23604663b8962b32ebf424fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:42.513861 2193 tablet_replica.cc:331] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc: stopping tablet replica
I20250115 02:52:42.514560 2193 raft_consensus.cc:2238] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:42.515172 2193 raft_consensus.cc:2267] T a1ac97a302e849ae8b6ac6d8214f347f P 324993df23604663b8962b32ebf424fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:42.518333 2193 tablet_replica.cc:331] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc: stopping tablet replica
I20250115 02:52:42.518986 2193 raft_consensus.cc:2238] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:42.519747 2193 raft_consensus.cc:2267] T d92e45d9f4e34b46ad5b9d3feea875bd P 324993df23604663b8962b32ebf424fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:42.541656 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:52:42.559556 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:52:42.597806 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:42.598323 2193 tablet_replica.cc:331] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c: stopping tablet replica
I20250115 02:52:42.598912 2193 raft_consensus.cc:2238] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:42.599815 2193 raft_consensus.cc:2267] T a1ac97a302e849ae8b6ac6d8214f347f P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:42.602082 2193 tablet_replica.cc:331] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c: stopping tablet replica
I20250115 02:52:42.602557 2193 raft_consensus.cc:2238] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:42.603005 2193 raft_consensus.cc:2267] T 5360b5e930e34cbbb0a84f0b662324bd P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:42.604619 2193 tablet_replica.cc:331] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c: stopping tablet replica
I20250115 02:52:42.605036 2193 raft_consensus.cc:2238] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:42.605402 2193 raft_consensus.cc:2267] T d92e45d9f4e34b46ad5b9d3feea875bd P acca59c4e7cb4a1c8bced20605d3d75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:42.629263 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:52:42.642351 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:52:42.680682 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:42.681300 2193 tablet_replica.cc:331] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3: stopping tablet replica
I20250115 02:52:42.681921 2193 raft_consensus.cc:2238] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:42.682361 2193 raft_consensus.cc:2267] T a1ac97a302e849ae8b6ac6d8214f347f P 33d4224a68324476b984325a0db0e8a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:42.684652 2193 tablet_replica.cc:331] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3: stopping tablet replica
I20250115 02:52:42.685364 2193 raft_consensus.cc:2238] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:42.686147 2193 raft_consensus.cc:2267] T 5360b5e930e34cbbb0a84f0b662324bd P 33d4224a68324476b984325a0db0e8a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:42.687794 2193 tablet_replica.cc:331] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3: stopping tablet replica
I20250115 02:52:42.688220 2193 raft_consensus.cc:2238] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:42.688602 2193 raft_consensus.cc:2267] T d92e45d9f4e34b46ad5b9d3feea875bd P 33d4224a68324476b984325a0db0e8a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:42.708158 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:52:42.721081 2193 master.cc:537] Master@127.2.36.126:39537 shutting down...
I20250115 02:52:42.741984 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:42.742453 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:42.742710 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 69bf2baaf0a9439f896922e654a99a3a: stopping tablet replica
W20250115 02:52:42.755501 5894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.36.126:39537: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20250115 02:52:42.757870 5894 consensus_peers.cc:487] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 -> Peer 69bf2baaf0a9439f896922e654a99a3a (127.2.36.126:39537): Couldn't send request to peer 69bf2baaf0a9439f896922e654a99a3a. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.126:39537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:42.760727 2193 master.cc:559] Master@127.2.36.126:39537 shutdown complete.
I20250115 02:52:42.774503 2193 master.cc:537] Master@127.2.36.125:40647 shutting down...
I20250115 02:52:42.791415 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:42.792214 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:42.792533 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P ea7c4e8ca85d4dcd8a06bc5f801dce19: stopping tablet replica
I20250115 02:52:42.811566 2193 master.cc:559] Master@127.2.36.125:40647 shutdown complete.
I20250115 02:52:42.824620 2193 master.cc:537] Master@127.2.36.124:34495 shutting down...
I20250115 02:52:42.840941 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:42.841420 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:42.841686 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 403273e9fdea49c3ba74e2340193a59a: stopping tablet replica
I20250115 02:52:42.859814 2193 master.cc:559] Master@127.2.36.124:34495 shutdown complete.
[ OK ] RemoteKsckTest.TestChecksumSnapshot (6962 ms)
[ RUN ] RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp
I20250115 02:52:42.891072 2193 test_util.cc:274] Using random seed: -981415125
I20250115 02:52:42.891990 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:46053,127.2.36.125:39803,127.2.36.124:45585
I20250115 02:52:42.893148 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:42.898722 6424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:42.899596 6425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:42.900298 6427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:42.901105 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:42.901968 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:42.902148 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:42.902278 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909562902262 us; error 0 us; skew 500 ppm
I20250115 02:52:42.902788 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:42.904898 2193 webserver.cc:458] Webserver started at http://127.2.36.126:34645/ using document root <none> and password file <none>
I20250115 02:52:42.905314 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:42.905474 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:42.905700 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:42.906778 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "b9e479ff36ff4e23a62d90bea40dbbf2"
format_stamp: "Formatted at 2025-01-15 02:52:42 on dist-test-slave-1lv2"
I20250115 02:52:42.910964 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250115 02:52:42.913851 6432 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:42.914489 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:52:42.914734 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "b9e479ff36ff4e23a62d90bea40dbbf2"
format_stamp: "Formatted at 2025-01-15 02:52:42 on dist-test-slave-1lv2"
I20250115 02:52:42.915014 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:42.938347 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:42.939486 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:42.974046 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:46053
I20250115 02:52:42.974135 6483 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:46053 every 8 connection(s)
I20250115 02:52:42.977573 6484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:42.977672 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:42.983728 6486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:42.984340 6487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:42.984833 6484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:42.986289 6489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:42.990010 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:42.990970 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:42.991150 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:42.991293 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909562991276 us; error 0 us; skew 500 ppm
I20250115 02:52:42.991789 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:42.994434 2193 webserver.cc:458] Webserver started at http://127.2.36.125:45263/ using document root <none> and password file <none>
I20250115 02:52:42.995019 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:42.995276 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:42.995585 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:42.996589 6484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:42.996945 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "e532bd0758d54ccca74891d5d555f832"
format_stamp: "Formatted at 2025-01-15 02:52:42 on dist-test-slave-1lv2"
W20250115 02:52:43.001335 6484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:39803: Network error: Client connection negotiation failed: client connection to 127.2.36.125:39803: connect: Connection refused (error 111)
I20250115 02:52:43.001351 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250115 02:52:43.004804 6497 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:43.005477 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:52:43.005743 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "e532bd0758d54ccca74891d5d555f832"
format_stamp: "Formatted at 2025-01-15 02:52:42 on dist-test-slave-1lv2"
I20250115 02:52:43.006006 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:43.019884 6484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } attempt: 1
I20250115 02:52:43.023484 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20250115 02:52:43.023803 6484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:39803: Network error: Client connection negotiation failed: client connection to 127.2.36.125:39803: connect: Connection refused (error 111)
I20250115 02:52:43.024567 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:43.058266 6484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } attempt: 2
I20250115 02:52:43.060017 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:39803
I20250115 02:52:43.060093 6548 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:39803 every 8 connection(s)
I20250115 02:52:43.065258 6550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:43.065613 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:43.071815 6484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:43.072666 6552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:43.074946 6550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:43.075968 6484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45585: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45585: connect: Connection refused (error 111)
W20250115 02:52:43.076439 6553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:43.082166 6556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:43.084007 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:43.084713 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:43.084911 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:43.085060 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909563085046 us; error 0 us; skew 500 ppm
I20250115 02:52:43.085625 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:43.088014 6550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:43.088510 2193 webserver.cc:458] Webserver started at http://127.2.36.124:37633/ using document root <none> and password file <none>
I20250115 02:52:43.088961 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:43.089141 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:43.089452 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:43.090834 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "7704565add4b41f08c2c7cd675fc98d8"
format_stamp: "Formatted at 2025-01-15 02:52:43 on dist-test-slave-1lv2"
I20250115 02:52:43.096086 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20250115 02:52:43.097146 6550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:43.101380 6561 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20250115 02:52:43.101742 6550 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45585: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45585: connect: Connection refused (error 111)
I20250115 02:52:43.102210 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250115 02:52:43.102468 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "7704565add4b41f08c2c7cd675fc98d8"
format_stamp: "Formatted at 2025-01-15 02:52:43 on dist-test-slave-1lv2"
I20250115 02:52:43.102692 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:43.121412 6484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } attempt: 1
I20250115 02:52:43.124874 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20250115 02:52:43.125368 6484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45585: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45585: connect: Connection refused (error 111)
I20250115 02:52:43.126026 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:43.160962 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:45585
I20250115 02:52:43.161041 6613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:45585 every 8 connection(s)
I20250115 02:52:43.163743 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:52:43.164608 6614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:43.165244 6550 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } attempt: 1
I20250115 02:52:43.168776 6484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } attempt: 2
I20250115 02:52:43.172932 6614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:43.183801 6550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832: Bootstrap starting.
I20250115 02:52:43.187121 6614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:43.187371 6484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2: Bootstrap starting.
I20250115 02:52:43.190044 6550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:43.193053 6484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:43.195609 6550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832: No bootstrap required, opened a new log
I20250115 02:52:43.198339 6614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:43.198638 6484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2: No bootstrap required, opened a new log
I20250115 02:52:43.198313 6550 raft_consensus.cc:357] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:43.199281 6550 raft_consensus.cc:383] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:43.199589 6550 raft_consensus.cc:738] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e532bd0758d54ccca74891d5d555f832, State: Initialized, Role: FOLLOWER
I20250115 02:52:43.200338 6550 consensus_queue.cc:260] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:43.201617 6484 raft_consensus.cc:357] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:43.203085 6484 raft_consensus.cc:383] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:43.203594 6484 raft_consensus.cc:738] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e479ff36ff4e23a62d90bea40dbbf2, State: Initialized, Role: FOLLOWER
I20250115 02:52:43.205060 6484 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:43.205690 6621 sys_catalog.cc:455] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:43.206498 6621 sys_catalog.cc:458] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:43.207442 6550 sys_catalog.cc:564] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:43.214082 6622 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:43.215106 6622 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:43.216707 6614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8: Bootstrap starting.
I20250115 02:52:43.216969 6484 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:43.223063 6614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8: Neither blocks nor log segments found. Creating new log.
W20250115 02:52:43.224241 6634 catalog_manager.cc:1559] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:43.224505 6634 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:43.228785 6614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8: No bootstrap required, opened a new log
I20250115 02:52:43.231796 6614 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:43.232720 6614 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:43.233027 6614 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7704565add4b41f08c2c7cd675fc98d8, State: Initialized, Role: FOLLOWER
I20250115 02:52:43.233415 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:52:43.233676 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:52:43.233783 6614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:43.235643 6646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:43.236415 6646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:43.236836 6614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [sys.catalog]: configured and running, proceeding with master startup.
W20250115 02:52:43.237179 6645 catalog_manager.cc:1559] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:43.237424 6645 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:43.247393 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:43.249059 6657 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:43.249348 6657 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:52:43.254346 6658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:43.255208 6659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:43.255931 6661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:43.256659 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:43.257368 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:43.257562 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:43.257714 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909563257696 us; error 0 us; skew 500 ppm
I20250115 02:52:43.258186 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:43.260371 2193 webserver.cc:458] Webserver started at http://127.2.36.65:43779/ using document root <none> and password file <none>
I20250115 02:52:43.260836 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:43.261056 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:43.261348 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:43.262425 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "8db93cc2942c46168f4d716539920329"
format_stamp: "Formatted at 2025-01-15 02:52:43 on dist-test-slave-1lv2"
I20250115 02:52:43.266366 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20250115 02:52:43.269300 6666 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:43.269981 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250115 02:52:43.270244 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "8db93cc2942c46168f4d716539920329"
format_stamp: "Formatted at 2025-01-15 02:52:43 on dist-test-slave-1lv2"
I20250115 02:52:43.270494 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250115 02:52:43.277249 6503 tablet.cc:2367] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250115 02:52:43.284474 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:43.285511 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:43.286842 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:43.288980 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:43.289152 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:43.289393 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:43.289535 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:43.314985 6621 raft_consensus.cc:491] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:43.315440 6621 raft_consensus.cc:513] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:43.317162 6621 leader_election.cc:290] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9e479ff36ff4e23a62d90bea40dbbf2 (127.2.36.126:46053), 7704565add4b41f08c2c7cd675fc98d8 (127.2.36.124:45585)
I20250115 02:52:43.317879 6459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e532bd0758d54ccca74891d5d555f832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" is_pre_election: true
I20250115 02:52:43.318259 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e532bd0758d54ccca74891d5d555f832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7704565add4b41f08c2c7cd675fc98d8" is_pre_election: true
I20250115 02:52:43.318603 6459 raft_consensus.cc:2463] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e532bd0758d54ccca74891d5d555f832 in term 0.
I20250115 02:52:43.318938 6589 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e532bd0758d54ccca74891d5d555f832 in term 0.
I20250115 02:52:43.319802 6500 leader_election.cc:304] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9e479ff36ff4e23a62d90bea40dbbf2, e532bd0758d54ccca74891d5d555f832; no voters:
I20250115 02:52:43.320538 6621 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:43.320808 6621 raft_consensus.cc:491] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:43.321111 6621 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:43.326093 6621 raft_consensus.cc:513] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:43.327639 6621 leader_election.cc:290] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [CANDIDATE]: Term 1 election: Requested vote from peers b9e479ff36ff4e23a62d90bea40dbbf2 (127.2.36.126:46053), 7704565add4b41f08c2c7cd675fc98d8 (127.2.36.124:45585)
I20250115 02:52:43.328727 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e532bd0758d54ccca74891d5d555f832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7704565add4b41f08c2c7cd675fc98d8"
I20250115 02:52:43.328716 6459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e532bd0758d54ccca74891d5d555f832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2"
I20250115 02:52:43.329452 6589 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:43.329595 6459 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:43.332468 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:44663
I20250115 02:52:43.332571 6728 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:44663 every 8 connection(s)
I20250115 02:52:43.335282 6589 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e532bd0758d54ccca74891d5d555f832 in term 1.
I20250115 02:52:43.336364 6500 leader_election.cc:304] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7704565add4b41f08c2c7cd675fc98d8, e532bd0758d54ccca74891d5d555f832; no voters:
I20250115 02:52:43.336916 6459 raft_consensus.cc:2463] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e532bd0758d54ccca74891d5d555f832 in term 1.
I20250115 02:52:43.337298 6621 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:43.345863 6621 raft_consensus.cc:695] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 1 LEADER]: Becoming Leader. State: Replica: e532bd0758d54ccca74891d5d555f832, State: Running, Role: LEADER
I20250115 02:52:43.346808 6621 consensus_queue.cc:237] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:43.350131 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:43.354801 6731 sys_catalog.cc:455] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e532bd0758d54ccca74891d5d555f832. Latest consensus state: current_term: 1 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:43.355618 6731 sys_catalog.cc:458] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:43.358347 6729 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45585
I20250115 02:52:43.358815 6729 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:43.359730 6729 heartbeater.cc:509] Master 127.2.36.124:45585 requested a full tablet report, sending...
I20250115 02:52:43.361016 6737 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
W20250115 02:52:43.361824 6738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:43.362519 6579 ts_manager.cc:194] Registered new tserver with Master: 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663)
W20250115 02:52:43.363845 6739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:43.368230 6730 heartbeater.cc:346] Connected to a master server at 127.2.36.125:39803
I20250115 02:52:43.368621 6730 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:43.369444 6730 heartbeater.cc:509] Master 127.2.36.125:39803 requested a full tablet report, sending...
I20250115 02:52:43.369596 6737 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:52:43.370537 2193 server_base.cc:1034] running on GCE node
W20250115 02:52:43.371600 6741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:43.371882 6514 ts_manager.cc:194] Registered new tserver with Master: 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663)
I20250115 02:52:43.373227 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:43.373493 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:43.373704 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909563373689 us; error 0 us; skew 500 ppm
I20250115 02:52:43.374480 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:43.380961 6589 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 1 FOLLOWER]: Refusing update from remote peer e532bd0758d54ccca74891d5d555f832: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:43.381238 6459 raft_consensus.cc:1270] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 1 FOLLOWER]: Refusing update from remote peer e532bd0758d54ccca74891d5d555f832: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:43.382805 2193 webserver.cc:458] Webserver started at http://127.2.36.66:33791/ using document root <none> and password file <none>
I20250115 02:52:43.383153 6733 heartbeater.cc:346] Connected to a master server at 127.2.36.126:46053
I20250115 02:52:43.383425 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:43.383512 6733 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:43.383626 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:43.384089 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:43.382992 6731 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 02:52:43.384429 6733 heartbeater.cc:509] Master 127.2.36.126:46053 requested a full tablet report, sending...
I20250115 02:52:43.385540 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "1ee170743349481cba07687363961f45"
format_stamp: "Formatted at 2025-01-15 02:52:43 on dist-test-slave-1lv2"
I20250115 02:52:43.385350 6621 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:43.387086 6449 ts_manager.cc:194] Registered new tserver with Master: 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663)
I20250115 02:52:43.391949 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250115 02:52:43.398648 6646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e532bd0758d54ccca74891d5d555f832. Latest consensus state: current_term: 1 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:43.399479 6646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:43.399657 6622 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e532bd0758d54ccca74891d5d555f832. Latest consensus state: current_term: 1 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:43.400374 6622 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:43.403362 6751 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:43.405294 2193 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.002s sys 0.004s
I20250115 02:52:43.405659 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "1ee170743349481cba07687363961f45"
format_stamp: "Formatted at 2025-01-15 02:52:43 on dist-test-slave-1lv2"
I20250115 02:52:43.406026 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:43.410508 6731 sys_catalog.cc:455] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:43.411378 6731 sys_catalog.cc:458] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:43.413161 6731 sys_catalog.cc:455] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:43.413933 6731 sys_catalog.cc:458] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:43.415550 6737 catalog_manager.cc:1348] Generated new cluster ID: 7a5854e1cee94722828261428e087b51
I20250115 02:52:43.415917 6737 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:52:43.417913 6646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:43.418664 6646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:43.424048 6622 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:43.424813 6622 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:43.437331 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:43.438915 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:43.441063 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:43.444943 6737 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:52:43.446828 6737 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:52:43.452979 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:43.453317 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:43.453644 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:43.453876 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:43.476632 6737 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832: Generated new TSK 0
I20250115 02:52:43.477193 6737 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:52:43.496392 6514 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42478
I20250115 02:52:43.509009 6730 heartbeater.cc:501] Master 127.2.36.125:39803 was elected leader, sending a full tablet report...
I20250115 02:52:43.517903 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:35963
I20250115 02:52:43.518002 6817 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:35963 every 8 connection(s)
I20250115 02:52:43.527315 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:43.537542 6818 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45585
I20250115 02:52:43.537946 6818 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:43.538837 6818 heartbeater.cc:509] Master 127.2.36.124:45585 requested a full tablet report, sending...
I20250115 02:52:43.542843 6579 ts_manager.cc:194] Registered new tserver with Master: 1ee170743349481cba07687363961f45 (127.2.36.66:35963)
W20250115 02:52:43.546612 6826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:43.553903 6828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:43.554911 6825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:43.554832 6820 heartbeater.cc:346] Connected to a master server at 127.2.36.126:46053
I20250115 02:52:43.555563 6819 heartbeater.cc:346] Connected to a master server at 127.2.36.125:39803
I20250115 02:52:43.555976 6820 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:43.556092 6819 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:43.556681 6820 heartbeater.cc:509] Master 127.2.36.126:46053 requested a full tablet report, sending...
I20250115 02:52:43.556860 6819 heartbeater.cc:509] Master 127.2.36.125:39803 requested a full tablet report, sending...
I20250115 02:52:43.559052 6513 ts_manager.cc:194] Registered new tserver with Master: 1ee170743349481cba07687363961f45 (127.2.36.66:35963)
I20250115 02:52:43.559222 6449 ts_manager.cc:194] Registered new tserver with Master: 1ee170743349481cba07687363961f45 (127.2.36.66:35963)
I20250115 02:52:43.560706 6513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42492
I20250115 02:52:43.561758 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:43.562803 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:43.563066 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:43.563233 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909563563220 us; error 0 us; skew 500 ppm
I20250115 02:52:43.563699 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:43.566274 2193 webserver.cc:458] Webserver started at http://127.2.36.67:46707/ using document root <none> and password file <none>
I20250115 02:52:43.566887 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:43.567095 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:43.567409 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:43.568801 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "a38e399727bc4503846b86b6aee45fa4"
format_stamp: "Formatted at 2025-01-15 02:52:43 on dist-test-slave-1lv2"
I20250115 02:52:43.573232 6819 heartbeater.cc:501] Master 127.2.36.125:39803 was elected leader, sending a full tablet report...
I20250115 02:52:43.573336 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250115 02:52:43.577664 6833 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:43.578514 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:52:43.578850 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "a38e399727bc4503846b86b6aee45fa4"
format_stamp: "Formatted at 2025-01-15 02:52:43 on dist-test-slave-1lv2"
I20250115 02:52:43.579110 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:43.592514 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:43.593930 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:43.595376 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:43.597494 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:43.597692 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:43.597910 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:43.598058 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:43.654736 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:37441
I20250115 02:52:43.654865 6895 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:37441 every 8 connection(s)
I20250115 02:52:43.680388 6896 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45585
I20250115 02:52:43.680771 6896 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:43.681572 6896 heartbeater.cc:509] Master 127.2.36.124:45585 requested a full tablet report, sending...
I20250115 02:52:43.684029 6897 heartbeater.cc:346] Connected to a master server at 127.2.36.125:39803
I20250115 02:52:43.684255 6578 ts_manager.cc:194] Registered new tserver with Master: a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:43.684362 6897 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:43.686488 6897 heartbeater.cc:509] Master 127.2.36.125:39803 requested a full tablet report, sending...
I20250115 02:52:43.688817 6513 ts_manager.cc:194] Registered new tserver with Master: a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:43.689235 6898 heartbeater.cc:346] Connected to a master server at 127.2.36.126:46053
I20250115 02:52:43.689623 6898 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:43.690402 6898 heartbeater.cc:509] Master 127.2.36.126:46053 requested a full tablet report, sending...
I20250115 02:52:43.690491 6513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42508
I20250115 02:52:43.692847 6448 ts_manager.cc:194] Registered new tserver with Master: a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:43.693015 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.029991849s
I20250115 02:52:43.703099 6897 heartbeater.cc:501] Master 127.2.36.125:39803 was elected leader, sending a full tablet report...
I20250115 02:52:43.740828 6513 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42522:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:52:43.742981 6513 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:52:43.820827 6783 tablet_service.cc:1467] Processing CreateTablet for tablet ece3e2ecf33a4c39bae992490a2f5cf0 (DEFAULT_TABLE table=ksck-test-table [id=653e85ea80884759bfa6d984105e31ac]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:43.822214 6783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ece3e2ecf33a4c39bae992490a2f5cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:43.836396 6694 tablet_service.cc:1467] Processing CreateTablet for tablet ece3e2ecf33a4c39bae992490a2f5cf0 (DEFAULT_TABLE table=ksck-test-table [id=653e85ea80884759bfa6d984105e31ac]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:43.837679 6694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ece3e2ecf33a4c39bae992490a2f5cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:43.839651 6693 tablet_service.cc:1467] Processing CreateTablet for tablet e9dace2ddc84456aa9f50378f3685951 (DEFAULT_TABLE table=ksck-test-table [id=653e85ea80884759bfa6d984105e31ac]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:43.845656 6693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9dace2ddc84456aa9f50378f3685951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:43.847756 6916 tablet_bootstrap.cc:492] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45: Bootstrap starting.
I20250115 02:52:43.839623 6782 tablet_service.cc:1467] Processing CreateTablet for tablet e9dace2ddc84456aa9f50378f3685951 (DEFAULT_TABLE table=ksck-test-table [id=653e85ea80884759bfa6d984105e31ac]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:43.840487 6692 tablet_service.cc:1467] Processing CreateTablet for tablet dca89dd97d744fdb996f4fea95acbf8a (DEFAULT_TABLE table=ksck-test-table [id=653e85ea80884759bfa6d984105e31ac]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:43.848846 6782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9dace2ddc84456aa9f50378f3685951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:43.848997 6692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dca89dd97d744fdb996f4fea95acbf8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:43.841001 6781 tablet_service.cc:1467] Processing CreateTablet for tablet dca89dd97d744fdb996f4fea95acbf8a (DEFAULT_TABLE table=ksck-test-table [id=653e85ea80884759bfa6d984105e31ac]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:43.860872 6781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dca89dd97d744fdb996f4fea95acbf8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:43.864715 6861 tablet_service.cc:1467] Processing CreateTablet for tablet ece3e2ecf33a4c39bae992490a2f5cf0 (DEFAULT_TABLE table=ksck-test-table [id=653e85ea80884759bfa6d984105e31ac]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:43.866021 6861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ece3e2ecf33a4c39bae992490a2f5cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:43.853739 6916 tablet_bootstrap.cc:654] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:43.883378 6860 tablet_service.cc:1467] Processing CreateTablet for tablet e9dace2ddc84456aa9f50378f3685951 (DEFAULT_TABLE table=ksck-test-table [id=653e85ea80884759bfa6d984105e31ac]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:43.884650 6860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9dace2ddc84456aa9f50378f3685951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:43.890834 6859 tablet_service.cc:1467] Processing CreateTablet for tablet dca89dd97d744fdb996f4fea95acbf8a (DEFAULT_TABLE table=ksck-test-table [id=653e85ea80884759bfa6d984105e31ac]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:43.892155 6859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dca89dd97d744fdb996f4fea95acbf8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:43.913671 6916 tablet_bootstrap.cc:492] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45: No bootstrap required, opened a new log
I20250115 02:52:43.914222 6916 ts_tablet_manager.cc:1397] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45: Time spent bootstrapping tablet: real 0.067s user 0.012s sys 0.000s
I20250115 02:52:43.917481 6916 raft_consensus.cc:357] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:43.918299 6916 raft_consensus.cc:383] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:43.914291 6918 tablet_bootstrap.cc:492] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4: Bootstrap starting.
I20250115 02:52:43.927942 6918 tablet_bootstrap.cc:654] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:43.920109 6919 tablet_bootstrap.cc:492] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329: Bootstrap starting.
I20250115 02:52:43.918584 6916 raft_consensus.cc:738] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee170743349481cba07687363961f45, State: Initialized, Role: FOLLOWER
I20250115 02:52:43.937501 6916 consensus_queue.cc:260] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:43.942795 6919 tablet_bootstrap.cc:654] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:43.952996 6916 ts_tablet_manager.cc:1428] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45: Time spent starting tablet: real 0.038s user 0.006s sys 0.001s
I20250115 02:52:43.954030 6916 tablet_bootstrap.cc:492] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45: Bootstrap starting.
I20250115 02:52:43.954320 6920 raft_consensus.cc:491] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:43.954805 6920 raft_consensus.cc:513] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:43.957386 6918 tablet_bootstrap.cc:492] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4: No bootstrap required, opened a new log
I20250115 02:52:43.957916 6918 ts_tablet_manager.cc:1397] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.005s
I20250115 02:52:43.960585 6918 raft_consensus.cc:357] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:43.961534 6918 raft_consensus.cc:383] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:43.961860 6918 raft_consensus.cc:738] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38e399727bc4503846b86b6aee45fa4, State: Initialized, Role: FOLLOWER
I20250115 02:52:43.962602 6918 consensus_queue.cc:260] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:43.964849 6919 tablet_bootstrap.cc:492] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329: No bootstrap required, opened a new log
I20250115 02:52:43.971649 6919 ts_tablet_manager.cc:1397] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329: Time spent bootstrapping tablet: real 0.052s user 0.016s sys 0.000s
I20250115 02:52:43.974383 6919 raft_consensus.cc:357] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:43.975101 6919 raft_consensus.cc:383] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:43.975386 6919 raft_consensus.cc:738] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db93cc2942c46168f4d716539920329, State: Initialized, Role: FOLLOWER
I20250115 02:52:43.976224 6919 consensus_queue.cc:260] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:43.993292 6919 ts_tablet_manager.cc:1428] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329: Time spent starting tablet: real 0.021s user 0.012s sys 0.000s
I20250115 02:52:43.994403 6916 tablet_bootstrap.cc:654] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:43.995785 6871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece3e2ecf33a4c39bae992490a2f5cf0" candidate_uuid: "1ee170743349481cba07687363961f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4" is_pre_election: true
I20250115 02:52:43.996542 6871 raft_consensus.cc:2463] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ee170743349481cba07687363961f45 in term 0.
I20250115 02:52:44.006958 6756 leader_election.cc:304] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee170743349481cba07687363961f45, a38e399727bc4503846b86b6aee45fa4; no voters:
I20250115 02:52:44.014966 6921 raft_consensus.cc:2798] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:44.015435 6921 raft_consensus.cc:491] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:44.015758 6921 raft_consensus.cc:3054] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:44.016338 6919 tablet_bootstrap.cc:492] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329: Bootstrap starting.
I20250115 02:52:44.034547 6918 ts_tablet_manager.cc:1428] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4: Time spent starting tablet: real 0.076s user 0.031s sys 0.039s
I20250115 02:52:44.037400 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece3e2ecf33a4c39bae992490a2f5cf0" candidate_uuid: "1ee170743349481cba07687363961f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db93cc2942c46168f4d716539920329" is_pre_election: true
W20250115 02:52:44.038715 6755 leader_election.cc:343] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:52:44.042742 6920 leader_election.cc:290] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441), 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663)
I20250115 02:52:44.043941 6918 tablet_bootstrap.cc:492] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4: Bootstrap starting.
I20250115 02:52:44.045611 6919 tablet_bootstrap.cc:654] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:44.049614 6918 tablet_bootstrap.cc:654] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:44.058629 6919 tablet_bootstrap.cc:492] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329: No bootstrap required, opened a new log
I20250115 02:52:44.059216 6919 ts_tablet_manager.cc:1397] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.011s
I20250115 02:52:44.062551 6916 tablet_bootstrap.cc:492] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45: No bootstrap required, opened a new log
I20250115 02:52:44.062613 6918 tablet_bootstrap.cc:492] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4: No bootstrap required, opened a new log
I20250115 02:52:44.063673 6918 ts_tablet_manager.cc:1397] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.010s
I20250115 02:52:44.064507 6919 raft_consensus.cc:357] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.065263 6919 raft_consensus.cc:383] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:44.065557 6919 raft_consensus.cc:738] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db93cc2942c46168f4d716539920329, State: Initialized, Role: FOLLOWER
I20250115 02:52:44.063158 6916 ts_tablet_manager.cc:1397] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45: Time spent bootstrapping tablet: real 0.109s user 0.001s sys 0.011s
I20250115 02:52:44.066586 6921 raft_consensus.cc:513] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:44.066557 6918 raft_consensus.cc:357] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.068406 6918 raft_consensus.cc:383] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:44.068735 6918 raft_consensus.cc:738] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38e399727bc4503846b86b6aee45fa4, State: Initialized, Role: FOLLOWER
I20250115 02:52:44.071346 6921 leader_election.cc:290] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 1 election: Requested vote from peers a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441), 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663)
I20250115 02:52:44.069339 6871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece3e2ecf33a4c39bae992490a2f5cf0" candidate_uuid: "1ee170743349481cba07687363961f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4"
I20250115 02:52:44.069643 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece3e2ecf33a4c39bae992490a2f5cf0" candidate_uuid: "1ee170743349481cba07687363961f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db93cc2942c46168f4d716539920329"
I20250115 02:52:44.072080 6871 raft_consensus.cc:3054] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:44.071821 6918 consensus_queue.cc:260] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
W20250115 02:52:44.072927 6755 leader_election.cc:343] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:52:44.074121 6918 ts_tablet_manager.cc:1428] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4: Time spent starting tablet: real 0.010s user 0.002s sys 0.002s
I20250115 02:52:44.075331 6918 tablet_bootstrap.cc:492] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4: Bootstrap starting.
I20250115 02:52:44.081398 6918 tablet_bootstrap.cc:654] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:44.081457 6871 raft_consensus.cc:2463] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ee170743349481cba07687363961f45 in term 1.
I20250115 02:52:44.082702 6756 leader_election.cc:304] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ee170743349481cba07687363961f45, a38e399727bc4503846b86b6aee45fa4; no voters: 8db93cc2942c46168f4d716539920329
I20250115 02:52:44.085389 6921 raft_consensus.cc:2798] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:44.091648 6921 raft_consensus.cc:695] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 1 LEADER]: Becoming Leader. State: Replica: 1ee170743349481cba07687363961f45, State: Running, Role: LEADER
I20250115 02:52:44.092574 6921 consensus_queue.cc:237] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:44.096277 6916 raft_consensus.cc:357] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.097101 6916 raft_consensus.cc:383] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:44.097409 6916 raft_consensus.cc:738] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee170743349481cba07687363961f45, State: Initialized, Role: FOLLOWER
I20250115 02:52:44.098235 6919 consensus_queue.cc:260] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.100708 6919 ts_tablet_manager.cc:1428] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20250115 02:52:44.101713 6919 tablet_bootstrap.cc:492] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329: Bootstrap starting.
I20250115 02:52:44.103067 6916 consensus_queue.cc:260] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.105000 6918 tablet_bootstrap.cc:492] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4: No bootstrap required, opened a new log
I20250115 02:52:44.105518 6918 ts_tablet_manager.cc:1397] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.002s
I20250115 02:52:44.106392 6916 ts_tablet_manager.cc:1428] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45: Time spent starting tablet: real 0.040s user 0.005s sys 0.001s
I20250115 02:52:44.107522 6916 tablet_bootstrap.cc:492] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45: Bootstrap starting.
I20250115 02:52:44.109386 6918 raft_consensus.cc:357] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.110117 6918 raft_consensus.cc:383] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:44.110409 6918 raft_consensus.cc:738] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38e399727bc4503846b86b6aee45fa4, State: Initialized, Role: FOLLOWER
I20250115 02:52:44.111080 6918 consensus_queue.cc:260] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.113344 6918 ts_tablet_manager.cc:1428] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250115 02:52:44.118256 6916 tablet_bootstrap.cc:654] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:44.119432 6919 tablet_bootstrap.cc:654] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:44.139734 6512 catalog_manager.cc:5526] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ee170743349481cba07687363961f45 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "1ee170743349481cba07687363961f45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:44.162855 6919 tablet_bootstrap.cc:492] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329: No bootstrap required, opened a new log
I20250115 02:52:44.163420 6919 ts_tablet_manager.cc:1397] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.005s
I20250115 02:52:44.167692 6919 raft_consensus.cc:357] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:44.168445 6919 raft_consensus.cc:383] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:44.168741 6919 raft_consensus.cc:738] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db93cc2942c46168f4d716539920329, State: Initialized, Role: FOLLOWER
I20250115 02:52:44.169592 6919 consensus_queue.cc:260] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:44.173504 6916 tablet_bootstrap.cc:492] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45: No bootstrap required, opened a new log
I20250115 02:52:44.174021 6916 ts_tablet_manager.cc:1397] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45: Time spent bootstrapping tablet: real 0.067s user 0.021s sys 0.037s
I20250115 02:52:44.175518 6919 ts_tablet_manager.cc:1428] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250115 02:52:44.176658 6916 raft_consensus.cc:357] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.177408 6916 raft_consensus.cc:383] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:44.177685 6916 raft_consensus.cc:738] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee170743349481cba07687363961f45, State: Initialized, Role: FOLLOWER
I20250115 02:52:44.178423 6916 consensus_queue.cc:260] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.187526 6916 ts_tablet_manager.cc:1428] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20250115 02:52:44.242350 6645 catalog_manager.cc:1260] Loaded cluster ID: 7a5854e1cee94722828261428e087b51
I20250115 02:52:44.242681 6645 catalog_manager.cc:1553] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2: loading cluster ID for follower catalog manager: success
I20250115 02:52:44.252137 6645 catalog_manager.cc:1575] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2: acquiring CA information for follower catalog manager: success
I20250115 02:52:44.253782 6657 catalog_manager.cc:1260] Loaded cluster ID: 7a5854e1cee94722828261428e087b51
I20250115 02:52:44.254081 6657 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8: loading cluster ID for follower catalog manager: success
I20250115 02:52:44.256996 6645 catalog_manager.cc:1603] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:44.262804 6925 raft_consensus.cc:491] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:44.263341 6925 raft_consensus.cc:513] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.265653 6925 leader_election.cc:290] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ee170743349481cba07687363961f45 (127.2.36.66:35963), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:44.285359 6657 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8: acquiring CA information for follower catalog manager: success
I20250115 02:52:44.289148 6793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca89dd97d744fdb996f4fea95acbf8a" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45" is_pre_election: true
I20250115 02:52:44.289489 6924 raft_consensus.cc:491] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:44.289958 6793 raft_consensus.cc:2463] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 0.
I20250115 02:52:44.290028 6924 raft_consensus.cc:513] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.294159 6924 leader_election.cc:290] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), 1ee170743349481cba07687363961f45 (127.2.36.66:35963)
I20250115 02:52:44.295423 6668 leader_election.cc:304] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329; no voters:
I20250115 02:52:44.296293 6925 raft_consensus.cc:2798] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:44.296705 6925 raft_consensus.cc:491] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:44.297030 6925 raft_consensus.cc:3054] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:44.309126 6871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca89dd97d744fdb996f4fea95acbf8a" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4" is_pre_election: true
I20250115 02:52:44.309866 6871 raft_consensus.cc:2463] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 0.
I20250115 02:52:44.310191 6925 raft_consensus.cc:513] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.312235 6925 leader_election.cc:290] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 1 election: Requested vote from peers 1ee170743349481cba07687363961f45 (127.2.36.66:35963), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:44.312856 6793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca89dd97d744fdb996f4fea95acbf8a" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45"
I20250115 02:52:44.313448 6793 raft_consensus.cc:3054] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:44.321904 6793 raft_consensus.cc:2463] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 1.
I20250115 02:52:44.324282 6871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca89dd97d744fdb996f4fea95acbf8a" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4"
I20250115 02:52:44.324906 6871 raft_consensus.cc:3054] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:44.329777 6657 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:44.331130 6668 leader_election.cc:304] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329; no voters:
I20250115 02:52:44.332036 6925 raft_consensus.cc:2798] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:44.335166 6871 raft_consensus.cc:2463] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 1.
I20250115 02:52:44.336887 6925 raft_consensus.cc:695] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 1 LEADER]: Becoming Leader. State: Replica: 8db93cc2942c46168f4d716539920329, State: Running, Role: LEADER
I20250115 02:52:44.337774 6925 consensus_queue.cc:237] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.339303 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca89dd97d744fdb996f4fea95acbf8a" candidate_uuid: "a38e399727bc4503846b86b6aee45fa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db93cc2942c46168f4d716539920329" is_pre_election: true
I20250115 02:52:44.354732 6793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca89dd97d744fdb996f4fea95acbf8a" candidate_uuid: "a38e399727bc4503846b86b6aee45fa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45" is_pre_election: true
I20250115 02:52:44.355576 6793 raft_consensus.cc:2388] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a38e399727bc4503846b86b6aee45fa4 in current term 1: Already voted for candidate 8db93cc2942c46168f4d716539920329 in this term.
I20250115 02:52:44.356465 6835 leader_election.cc:304] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a38e399727bc4503846b86b6aee45fa4; no voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329
I20250115 02:52:44.356488 6514 catalog_manager.cc:5526] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8db93cc2942c46168f4d716539920329 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "8db93cc2942c46168f4d716539920329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:44.357405 6924 raft_consensus.cc:2743] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:52:44.358120 6937 raft_consensus.cc:491] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:44.358593 6937 raft_consensus.cc:513] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.363112 6793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45" is_pre_election: true
I20250115 02:52:44.363768 6793 raft_consensus.cc:2463] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 0.
I20250115 02:52:44.368196 6668 leader_election.cc:304] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329; no voters:
I20250115 02:52:44.368254 6937 leader_election.cc:290] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ee170743349481cba07687363961f45 (127.2.36.66:35963), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:44.369076 6937 raft_consensus.cc:2798] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:44.369470 6937 raft_consensus.cc:491] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:44.369853 6937 raft_consensus.cc:3054] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:44.369804 6871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4" is_pre_election: true
I20250115 02:52:44.370616 6871 raft_consensus.cc:2463] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 0.
I20250115 02:52:44.377096 6937 raft_consensus.cc:513] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.379833 6937 leader_election.cc:290] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 1 election: Requested vote from peers 1ee170743349481cba07687363961f45 (127.2.36.66:35963), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:44.380944 6871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4"
I20250115 02:52:44.381553 6871 raft_consensus.cc:3054] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:44.383492 6793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45"
I20250115 02:52:44.384102 6793 raft_consensus.cc:3054] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:44.388296 6871 raft_consensus.cc:2463] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 1.
I20250115 02:52:44.389330 6668 leader_election.cc:304] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8db93cc2942c46168f4d716539920329, a38e399727bc4503846b86b6aee45fa4; no voters:
I20250115 02:52:44.390180 6937 raft_consensus.cc:2798] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:44.390637 6937 raft_consensus.cc:695] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 1 LEADER]: Becoming Leader. State: Replica: 8db93cc2942c46168f4d716539920329, State: Running, Role: LEADER
I20250115 02:52:44.391458 6937 consensus_queue.cc:237] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:44.410645 6512 catalog_manager.cc:5526] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8db93cc2942c46168f4d716539920329 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "8db93cc2942c46168f4d716539920329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:44.413093 6793 raft_consensus.cc:2463] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 1.
I20250115 02:52:44.455228 6937 raft_consensus.cc:491] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:44.455789 6937 raft_consensus.cc:513] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:44.457737 6937 leader_election.cc:290] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ee170743349481cba07687363961f45 (127.2.36.66:35963), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:44.458643 6793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece3e2ecf33a4c39bae992490a2f5cf0" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45" is_pre_election: true
I20250115 02:52:44.459031 6871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece3e2ecf33a4c39bae992490a2f5cf0" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4" is_pre_election: true
I20250115 02:52:44.459843 6871 raft_consensus.cc:2388] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8db93cc2942c46168f4d716539920329 in current term 1: Already voted for candidate 1ee170743349481cba07687363961f45 in this term.
I20250115 02:52:44.460692 6668 leader_election.cc:304] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8db93cc2942c46168f4d716539920329; no voters: 1ee170743349481cba07687363961f45, a38e399727bc4503846b86b6aee45fa4
I20250115 02:52:44.461820 6937 raft_consensus.cc:3054] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:44.468524 6937 raft_consensus.cc:2743] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:52:44.536837 6704 raft_consensus.cc:1270] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Refusing update from remote peer 1ee170743349481cba07687363961f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:44.538462 6921 consensus_queue.cc:1035] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:44.538455 6871 raft_consensus.cc:1270] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Refusing update from remote peer 1ee170743349481cba07687363961f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:44.543315 6921 consensus_queue.cc:1035] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:44.562335 6950 mvcc.cc:204] Tried to move back new op lower bound from 7114381576330391552 to 7114381574584348672. Current Snapshot: MvccSnapshot[applied={T|T < 7114381576330391552}]
I20250115 02:52:44.786975 6937 consensus_queue.cc:1035] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:44.832247 6871 raft_consensus.cc:1270] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Refusing update from remote peer 8db93cc2942c46168f4d716539920329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:44.835731 6960 consensus_queue.cc:1035] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:44.836025 6793 raft_consensus.cc:1270] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Refusing update from remote peer 8db93cc2942c46168f4d716539920329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:44.841715 6937 consensus_queue.cc:1035] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:44.883821 6937 consensus_queue.cc:1035] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 02:52:46.760258 6668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.36.66:35963, user_credentials={real_user=slave}} blocked reactor thread for 666135us
W20250115 02:52:47.038292 6668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.36.67:37441, user_credentials={real_user=slave}} blocked reactor thread for 276924us
I20250115 02:52:47.607342 6972 raft_consensus.cc:491] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8db93cc2942c46168f4d716539920329)
I20250115 02:52:47.608105 6972 raft_consensus.cc:513] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:47.611121 6972 leader_election.cc:290] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), 1ee170743349481cba07687363961f45 (127.2.36.66:35963)
I20250115 02:52:47.612286 6578 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50302 (request call id 152) took 1575 ms. Trace:
I20250115 02:52:47.612717 6578 rpcz_store.cc:276] 0115 02:52:46.036882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:52:46.041064 (+ 4182us) service_pool.cc:225] Handling call
0115 02:52:47.612209 (+1571145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 02:52:47.674080 6974 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e532bd0758d54ccca74891d5d555f832)
I20250115 02:52:47.674707 6974 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:47.706060 6973 raft_consensus.cc:491] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8db93cc2942c46168f4d716539920329)
I20250115 02:52:47.706739 6973 raft_consensus.cc:513] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:47.709951 6973 leader_election.cc:290] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:47.725821 6974 leader_election.cc:290] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9e479ff36ff4e23a62d90bea40dbbf2 (127.2.36.126:46053), e532bd0758d54ccca74891d5d555f832 (127.2.36.125:39803)
W20250115 02:52:47.900867 6668 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.812s user 0.002s sys 0.000s
W20250115 02:52:47.901331 6668 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.812s user 0.002s sys 0.000s
W20250115 02:52:47.901703 6668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.36.66:35963, user_credentials={real_user=slave}} blocked reactor thread for 812608us
W20250115 02:52:48.145354 6814 debug-util.cc:398] Leaking SignalData structure 0x7b08002466a0 after lost signal to thread 2196
I20250115 02:52:48.597285 6976 raft_consensus.cc:491] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ee170743349481cba07687363961f45)
I20250115 02:52:48.597960 6976 raft_consensus.cc:513] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:48.600364 6976 leader_election.cc:290] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ee170743349481cba07687363961f45 (127.2.36.66:35963), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:49.172080 6977 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e532bd0758d54ccca74891d5d555f832)
I20250115 02:52:49.173571 6977 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:49.176843 6589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50290 (request call id 22) took 1499 ms. Trace:
I20250115 02:52:49.177488 6589 rpcz_store.cc:276] 0115 02:52:47.677168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:52:49.174624 (+1497456us) service_pool.cc:225] Handling call
0115 02:52:49.175377 (+ 753us) raft_consensus.cc:1466] Updating replica for 0 ops
0115 02:52:49.175940 (+ 563us) raft_consensus.cc:1529] Early marking committed up to index 0
0115 02:52:49.176014 (+ 74us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0115 02:52:49.176106 (+ 92us) raft_consensus.cc:1640] Marking committed up to 9
0115 02:52:49.176216 (+ 110us) raft_consensus.cc:1692] Filling consensus response to leader.
0115 02:52:49.176319 (+ 103us) raft_consensus.cc:1686] UpdateReplicas() finished
0115 02:52:49.176586 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 02:52:49.183223 6459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7704565add4b41f08c2c7cd675fc98d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" is_pre_election: true
I20250115 02:52:49.185001 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7704565add4b41f08c2c7cd675fc98d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e532bd0758d54ccca74891d5d555f832" is_pre_election: true
I20250115 02:52:49.190165 6512 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42508 (request call id 127) took 3121 ms. Trace:
I20250115 02:52:49.190594 6512 rpcz_store.cc:276] 0115 02:52:46.068122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:52:46.070065 (+ 1943us) service_pool.cc:225] Handling call
0115 02:52:49.189903 (+3119838us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":0}
I20250115 02:52:49.197422 6459 raft_consensus.cc:2463] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7704565add4b41f08c2c7cd675fc98d8 in term 1.
I20250115 02:52:49.198596 6448 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53906 (request call id 143) took 3139 ms. Trace:
I20250115 02:52:49.200084 6448 rpcz_store.cc:276] 0115 02:52:46.430969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:52:49.196457 (+2765488us) service_pool.cc:225] Handling call
0115 02:52:49.198375 (+ 1918us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 02:52:49.205840 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece3e2ecf33a4c39bae992490a2f5cf0" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45" is_pre_election: true
I20250115 02:52:49.214131 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e532bd0758d54ccca74891d5d555f832" is_pre_election: true
I20250115 02:52:49.214323 6977 leader_election.cc:290] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e532bd0758d54ccca74891d5d555f832 (127.2.36.125:39803), 7704565add4b41f08c2c7cd675fc98d8 (127.2.36.124:45585)
I20250115 02:52:49.215276 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "7704565add4b41f08c2c7cd675fc98d8" is_pre_election: true
I20250115 02:52:49.220054 6435 leader_election.cc:304] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9e479ff36ff4e23a62d90bea40dbbf2; no voters: 7704565add4b41f08c2c7cd675fc98d8, e532bd0758d54ccca74891d5d555f832
I20250115 02:52:49.221861 6977 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 02:52:49.230844 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca89dd97d744fdb996f4fea95acbf8a" candidate_uuid: "a38e399727bc4503846b86b6aee45fa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "8db93cc2942c46168f4d716539920329" is_pre_election: true
I20250115 02:52:49.241305 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca89dd97d744fdb996f4fea95acbf8a" candidate_uuid: "1ee170743349481cba07687363961f45" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "8db93cc2942c46168f4d716539920329" is_pre_election: true
I20250115 02:52:49.244921 6980 raft_consensus.cc:491] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8db93cc2942c46168f4d716539920329)
I20250115 02:52:49.245826 6980 raft_consensus.cc:513] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:49.248453 6980 leader_election.cc:290] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), 1ee170743349481cba07687363961f45 (127.2.36.66:35963)
I20250115 02:52:49.249109 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece3e2ecf33a4c39bae992490a2f5cf0" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4" is_pre_election: true
I20250115 02:52:49.249775 6868 raft_consensus.cc:2463] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 1.
I20250115 02:52:49.251155 6668 leader_election.cc:304] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8db93cc2942c46168f4d716539920329, a38e399727bc4503846b86b6aee45fa4; no voters: 1ee170743349481cba07687363961f45
I20250115 02:52:49.252295 6969 raft_consensus.cc:2798] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:52:49.252779 6969 raft_consensus.cc:491] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ee170743349481cba07687363961f45)
I20250115 02:52:49.252544 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "a38e399727bc4503846b86b6aee45fa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8db93cc2942c46168f4d716539920329" is_pre_election: true
I20250115 02:52:49.253170 6969 raft_consensus.cc:3054] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:52:49.262656 6979 raft_consensus.cc:491] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8db93cc2942c46168f4d716539920329)
I20250115 02:52:49.263489 6979 raft_consensus.cc:513] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:49.264994 6513 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42478 (request call id 148) took 2743 ms. Trace:
I20250115 02:52:49.265982 6513 rpcz_store.cc:276] 0115 02:52:46.521954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:52:46.786783 (+264829us) service_pool.cc:225] Handling call
0115 02:52:49.263518 (+2476735us) spinlock_profiling.cc:243] Waited 2.39 s on lock 0x7b540020de28. stack: 00007f5a18a69f63 00007f5a18a69e60 00007f5a183bb584 00007f5a2242cff4 00007f5a2242cf89 00007f5a21f67521 00007f5a21f625dc 00007f5a2149edd4 00007f5a21596dee 00007f5a1fbfefc4 00007f5a1fbfef52 00007f5a1fbfee81 00007f5a1fbfedfc 00007f5a1fbfe0b2 00007f5a198d3dfc 00007f5a198d3236
0115 02:52:49.264831 (+ 1313us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":0,"spinlock_wait_cycles":5504214784}
I20250115 02:52:49.271337 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "1ee170743349481cba07687363961f45" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8db93cc2942c46168f4d716539920329" is_pre_election: true
I20250115 02:52:49.272687 6793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca89dd97d744fdb996f4fea95acbf8a" candidate_uuid: "a38e399727bc4503846b86b6aee45fa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45" is_pre_election: true
I20250115 02:52:49.274173 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "a38e399727bc4503846b86b6aee45fa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45" is_pre_election: true
I20250115 02:52:49.274528 6979 leader_election.cc:290] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:49.274873 6791 raft_consensus.cc:2463] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a38e399727bc4503846b86b6aee45fa4 in term 1.
I20250115 02:52:49.274672 6969 raft_consensus.cc:513] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:49.274852 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "1ee170743349481cba07687363961f45" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4" is_pre_election: true
I20250115 02:52:49.275889 6868 raft_consensus.cc:2463] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ee170743349481cba07687363961f45 in term 1.
I20250115 02:52:49.276990 6756 leader_election.cc:304] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ee170743349481cba07687363961f45, a38e399727bc4503846b86b6aee45fa4; no voters: 8db93cc2942c46168f4d716539920329
I20250115 02:52:49.277664 6835 leader_election.cc:304] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a38e399727bc4503846b86b6aee45fa4; no voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329
I20250115 02:52:49.278018 6969 leader_election.cc:290] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 2 election: Requested vote from peers 1ee170743349481cba07687363961f45 (127.2.36.66:35963), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:49.279438 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece3e2ecf33a4c39bae992490a2f5cf0" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45"
I20250115 02:52:49.279438 6835 leader_election.cc:304] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ee170743349481cba07687363961f45, a38e399727bc4503846b86b6aee45fa4; no voters: 8db93cc2942c46168f4d716539920329
I20250115 02:52:49.279848 6980 raft_consensus.cc:2743] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 02:52:49.280671 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece3e2ecf33a4c39bae992490a2f5cf0" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4"
I20250115 02:52:49.281265 6868 raft_consensus.cc:3054] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:52:49.281905 6979 raft_consensus.cc:2798] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:52:49.282339 6979 raft_consensus.cc:491] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8db93cc2942c46168f4d716539920329)
I20250115 02:52:49.282800 6979 raft_consensus.cc:3054] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:52:49.282590 6980 raft_consensus.cc:2798] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:52:49.286561 6980 raft_consensus.cc:491] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8db93cc2942c46168f4d716539920329)
I20250115 02:52:49.287042 6980 raft_consensus.cc:3054] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:52:49.292610 6979 raft_consensus.cc:513] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:49.296074 6868 raft_consensus.cc:2463] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 2.
I20250115 02:52:49.297677 6980 raft_consensus.cc:513] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:49.298179 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "1ee170743349481cba07687363961f45" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8db93cc2942c46168f4d716539920329"
I20250115 02:52:49.299422 6668 leader_election.cc:304] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8db93cc2942c46168f4d716539920329, a38e399727bc4503846b86b6aee45fa4; no voters: 1ee170743349481cba07687363961f45
I20250115 02:52:49.300637 6980 leader_election.cc:290] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 election: Requested vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), 1ee170743349481cba07687363961f45 (127.2.36.66:35963)
I20250115 02:52:49.300835 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "a38e399727bc4503846b86b6aee45fa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45"
I20250115 02:52:49.299436 6871 raft_consensus.cc:1235] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Rejecting Update request from peer 1ee170743349481cba07687363961f45 for earlier term 1. Current term is 2. Ops: []
W20250115 02:52:49.301710 6814 debug-util.cc:398] Leaking SignalData structure 0x7b0800243660 after lost signal to thread 6483
I20250115 02:52:49.301908 6791 raft_consensus.cc:2388] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a38e399727bc4503846b86b6aee45fa4 in current term 2: Already voted for candidate 1ee170743349481cba07687363961f45 in this term.
I20250115 02:52:49.302568 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "a38e399727bc4503846b86b6aee45fa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8db93cc2942c46168f4d716539920329"
I20250115 02:52:49.302902 6867 raft_consensus.cc:1235] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Rejecting Update request from peer 8db93cc2942c46168f4d716539920329 for earlier term 1. Current term is 2. Ops: []
I20250115 02:52:49.302089 6871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45952 (request call id 24) took 1629 ms. Trace:
W20250115 02:52:49.292521 6870 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45952 (request call id 25) took 1582 ms (client timeout 1582 ms). Trace:
I20250115 02:52:49.304487 6871 rpcz_store.cc:276] 0115 02:52:47.673172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:52:49.298690 (+1625518us) service_pool.cc:225] Handling call
0115 02:52:49.299104 (+ 414us) raft_consensus.cc:1466] Updating replica for 0 ops
0115 02:52:49.301814 (+ 2710us) raft_consensus.cc:1692] Filling consensus response to leader.
0115 02:52:49.302017 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250115 02:52:49.304703 6870 rpcz_store.cc:269] 0115 02:52:47.709780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:52:49.292067 (+1582287us) service_pool.cc:212] Skipping call since client already timed out
0115 02:52:49.292310 (+ 243us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250115 02:52:49.306169 6834 leader_election.cc:304] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a38e399727bc4503846b86b6aee45fa4; no voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329
I20250115 02:52:49.308703 6980 raft_consensus.cc:2743] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250115 02:52:49.310540 6979 leader_election.cc:290] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 2 election: Requested vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:49.311383 6979 consensus_queue.cc:1046] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 }, Status: INVALID_TERM, Last received: 1.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20250115 02:52:49.322796 6814 debug-util.cc:398] Leaking SignalData structure 0x7b080008a560 after lost signal to thread 6548
W20250115 02:52:49.325872 6814 debug-util.cc:398] Leaking SignalData structure 0x7b08001617a0 after lost signal to thread 6613
W20250115 02:52:49.324389 6756 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Timed out: Call waited in the queue past client deadline [suppressed 6 similar messages]
I20250115 02:52:49.393991 6988 raft_consensus.cc:3049] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 1 LEADER]: Stepping down as leader of term 1
I20250115 02:52:49.394476 6988 raft_consensus.cc:738] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1ee170743349481cba07687363961f45, State: Running, Role: LEADER
W20250115 02:52:49.385146 6668 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.36.67:37441, user_credentials={real_user=slave}} blocked reactor thread for 75790.2us
I20250115 02:52:49.398382 6986 consensus_queue.cc:1046] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 }, Status: INVALID_TERM, Last received: 1.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.006s
I20250115 02:52:49.404378 6514 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42492 (request call id 138) took 2882 ms. Trace:
I20250115 02:52:49.396114 6988 consensus_queue.cc:260] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:49.417176 6988 raft_consensus.cc:3054] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:52:49.414657 6986 raft_consensus.cc:3049] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 1 LEADER]: Stepping down as leader of term 1
I20250115 02:52:49.424775 6986 raft_consensus.cc:738] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8db93cc2942c46168f4d716539920329, State: Running, Role: LEADER
I20250115 02:52:49.425776 6986 consensus_queue.cc:260] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:49.427021 6986 raft_consensus.cc:3054] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:52:49.398672 6964 raft_consensus.cc:2798] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 2 FOLLOWER]: Leader election won for term 2
I20250115 02:52:49.429836 6964 raft_consensus.cc:695] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 2 LEADER]: Becoming Leader. State: Replica: 8db93cc2942c46168f4d716539920329, State: Running, Role: LEADER
I20250115 02:52:49.430830 6964 consensus_queue.cc:237] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:49.412071 6514 rpcz_store.cc:276] 0115 02:52:46.521610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:52:47.645286 (+1123676us) service_pool.cc:225] Handling call
0115 02:52:49.396851 (+1751565us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b540020de28. stack: 00007f5a18a69f63 00007f5a18a69e60 00007f5a183bb584 00007f5a2242cff4 00007f5a2242cf89 00007f5a21f67521 00007f5a21f625dc 00007f5a2149edd4 00007f5a21596dee 00007f5a1fbfefc4 00007f5a1fbfef52 00007f5a1fbfee81 00007f5a1fbfedfc 00007f5a1fbfe0b2 00007f5a198d3dfc 00007f5a198d3236
0115 02:52:49.404208 (+ 7357us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":0,"spinlock_wait_cycles":3532048256}
I20250115 02:52:49.445652 6992 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58630 (ReqId={client: 36ea471d53d9450db7cb6432ee230433, seq_no=67, attempt_no=0}) took 2559 ms. Trace:
W20250115 02:52:49.449640 6814 debug-util.cc:398] Leaking SignalData structure 0x7b08000637e0 after lost signal to thread 6728
W20250115 02:52:49.450997 6814 debug-util.cc:398] Leaking SignalData structure 0x7b080021e4e0 after lost signal to thread 6817
W20250115 02:52:49.461128 6814 debug-util.cc:398] Leaking SignalData structure 0x7b08001c9ea0 after lost signal to thread 6895
I20250115 02:52:49.467347 6565 leader_election.cc:304] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7704565add4b41f08c2c7cd675fc98d8, b9e479ff36ff4e23a62d90bea40dbbf2; no voters: e532bd0758d54ccca74891d5d555f832
W20250115 02:52:49.488958 6756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.36.67:37441, user_credentials={real_user=slave}} blocked reactor thread for 164627us
I20250115 02:52:49.491814 6870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "1ee170743349481cba07687363961f45" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4"
I20250115 02:52:49.492663 6870 raft_consensus.cc:2388] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ee170743349481cba07687363961f45 in current term 2: Already voted for candidate a38e399727bc4503846b86b6aee45fa4 in this term.
I20250115 02:52:49.446058 6992 rpcz_store.cc:276] 0115 02:52:46.886417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:52:46.886571 (+ 154us) service_pool.cc:225] Handling call
0115 02:52:49.445408 (+2558837us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0115 02:52:47.069052 (+ 0us) write_op.cc:183] PREPARE: starting on tablet dca89dd97d744fdb996f4fea95acbf8a
0115 02:52:47.069474 (+ 422us) write_op.cc:432] Acquiring schema lock in shared mode
0115 02:52:47.069508 (+ 34us) write_op.cc:435] Acquired schema lock
0115 02:52:47.069525 (+ 17us) tablet.cc:659] Decoding operations
0115 02:52:47.069782 (+ 257us) write_op.cc:620] Acquiring the partition lock for write op
0115 02:52:47.069861 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0115 02:52:47.069901 (+ 40us) tablet.cc:682] Acquiring locks for 1 operations
0115 02:52:47.070054 (+ 153us) tablet.cc:698] Row locks acquired
0115 02:52:47.070067 (+ 13us) write_op.cc:260] PREPARE: finished
0115 02:52:47.070248 (+ 181us) write_op.cc:270] Start()
0115 02:52:47.070357 (+ 109us) write_op.cc:276] Timestamp: P: 1736909567070200 usec, L: 0
0115 02:52:47.070376 (+ 19us) op_driver.cc:348] REPLICATION: starting
0115 02:52:47.905396 (+835020us) log.cc:844] Serialized 299 byte log entry
0115 02:52:49.316452 (+1411056us) op_driver.cc:464] REPLICATION: finished
0115 02:52:49.421644 (+105192us) write_op.cc:301] APPLY: starting
0115 02:52:49.421737 (+ 93us) tablet.cc:1364] starting BulkCheckPresence
0115 02:52:49.421880 (+ 143us) tablet.cc:1367] finished BulkCheckPresence
0115 02:52:49.421999 (+ 119us) tablet.cc:1369] starting ApplyRowOperation cycle
0115 02:52:49.422338 (+ 339us) tablet.cc:1380] finished ApplyRowOperation cycle
0115 02:52:49.422543 (+ 205us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0115 02:52:49.422586 (+ 43us) write_op.cc:312] APPLY: finished
0115 02:52:49.422808 (+ 222us) log.cc:844] Serialized 22 byte log entry
0115 02:52:49.444873 (+ 22065us) write_op.cc:489] Releasing partition, row and schema locks
0115 02:52:49.444977 (+ 104us) write_op.cc:454] Released schema lock
0115 02:52:49.445011 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":104894,"num_ops":1,"prepare.queue_time_us":180944,"prepare.run_cpu_time_us":5069,"prepare.run_wall_time_us":1028971,"raft.queue_time_us":2678,"raft.run_cpu_time_us":1830,"raft.run_wall_time_us":1979,"replication_time_us":2245918,"thread_start_us":474373,"threads_started":5,"wal-append.queue_time_us":179517,"wal-append.run_cpu_time_us":1946,"wal-append.run_wall_time_us":1144959}]]}
I20250115 02:52:49.498282 6756 leader_election.cc:304] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ee170743349481cba07687363961f45; no voters: 8db93cc2942c46168f4d716539920329, a38e399727bc4503846b86b6aee45fa4
I20250115 02:52:49.499373 6979 raft_consensus.cc:2743] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250115 02:52:49.500659 6870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca89dd97d744fdb996f4fea95acbf8a" candidate_uuid: "1ee170743349481cba07687363961f45" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4" is_pre_election: true
I20250115 02:52:49.502476 6756 leader_election.cc:304] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ee170743349481cba07687363961f45; no voters: 8db93cc2942c46168f4d716539920329, a38e399727bc4503846b86b6aee45fa4
I20250115 02:52:49.503389 6979 raft_consensus.cc:2743] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 02:52:49.512158 6792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52030 (request call id 113) took 1425 ms. Trace:
I20250115 02:52:49.512653 6792 rpcz_store.cc:276] 0115 02:52:48.068296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:52:49.257095 (+1188799us) service_pool.cc:225] Handling call
0115 02:52:49.257867 (+ 772us) raft_consensus.cc:1466] Updating replica for 1 ops
0115 02:52:49.258545 (+ 678us) raft_consensus.cc:1529] Early marking committed up to index 48
0115 02:52:49.305957 (+ 47412us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0115 02:52:49.306386 (+ 429us) write_op.cc:270] Start()
0115 02:52:49.306509 (+ 123us) write_op.cc:276] Timestamp: P: 1736909567070200 usec, L: 0
0115 02:52:49.472264 (+165755us) log.cc:844] Serialized 299 byte log entry
0115 02:52:49.472411 (+ 147us) raft_consensus.cc:1640] Marking committed up to 48
0115 02:52:49.472448 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0115 02:52:49.473722 (+ 1274us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0115 02:52:49.484101 (+ 10379us) raft_consensus.cc:1680] finished
0115 02:52:49.484122 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0115 02:52:49.493657 (+ 9535us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0115 02:52:49.496544 (+ 0us) write_op.cc:183] PREPARE: starting on tablet dca89dd97d744fdb996f4fea95acbf8a
0115 02:52:49.496915 (+ 371us) write_op.cc:432] Acquiring schema lock in shared mode
0115 02:52:49.496934 (+ 19us) write_op.cc:435] Acquired schema lock
0115 02:52:49.496941 (+ 7us) tablet.cc:659] Decoding operations
0115 02:52:49.497137 (+ 196us) write_op.cc:620] Acquiring the partition lock for write op
0115 02:52:49.497200 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0115 02:52:49.497225 (+ 25us) tablet.cc:682] Acquiring locks for 1 operations
0115 02:52:49.497384 (+ 159us) tablet.cc:698] Row locks acquired
0115 02:52:49.497420 (+ 36us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":29,"prepare.queue_time_us":189191,"prepare.run_cpu_time_us":1169,"prepare.run_wall_time_us":1169,"thread_start_us":163736,"threads_started":1}]]}
W20250115 02:52:49.515704 6565 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.36.126:46053, user_credentials={real_user=slave}} blocked reactor thread for 49869.2us
I20250115 02:52:49.517648 6996 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:52:49.518111 6996 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e532bd0758d54ccca74891d5d555f832)
I20250115 02:52:49.518476 6996 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:52:49.530865 6996 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:49.534211 6996 leader_election.cc:290] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [CANDIDATE]: Term 2 election: Requested vote from peers b9e479ff36ff4e23a62d90bea40dbbf2 (127.2.36.126:46053), e532bd0758d54ccca74891d5d555f832 (127.2.36.125:39803)
I20250115 02:52:49.534397 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7704565add4b41f08c2c7cd675fc98d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e532bd0758d54ccca74891d5d555f832"
I20250115 02:52:49.535501 6459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7704565add4b41f08c2c7cd675fc98d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2"
I20250115 02:52:49.536317 6589 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 2 FOLLOWER]: Rejecting Update request from peer e532bd0758d54ccca74891d5d555f832 for earlier term 1. Current term is 2. Ops: []
I20250115 02:52:49.537830 6621 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 }, Status: INVALID_TERM, Last received: 1.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250115 02:52:49.538239 6565 leader_election.cc:304] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7704565add4b41f08c2c7cd675fc98d8; no voters: b9e479ff36ff4e23a62d90bea40dbbf2, e532bd0758d54ccca74891d5d555f832
I20250115 02:52:49.538842 6621 raft_consensus.cc:3049] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 1 LEADER]: Stepping down as leader of term 1
I20250115 02:52:49.539203 6621 raft_consensus.cc:738] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e532bd0758d54ccca74891d5d555f832, State: Running, Role: LEADER
I20250115 02:52:49.539986 6621 consensus_queue.cc:260] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:49.541254 6621 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:52:49.546818 6996 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250115 02:52:49.546535 6514 catalog_manager.cc:5526] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 reported cstate change: term changed from 1 to 2, leader changed from 1ee170743349481cba07687363961f45 (127.2.36.66) to 8db93cc2942c46168f4d716539920329 (127.2.36.65). New cstate: current_term: 2 leader_uuid: "8db93cc2942c46168f4d716539920329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } health_report { overall_health: HEALTHY } } }
E20250115 02:52:49.554841 6514 catalog_manager.cc:5681] Error updating tablets from 8db93cc2942c46168f4d716539920329: Illegal state: replica e532bd0758d54ccca74891d5d555f832 is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "dca89dd97d744fdb996f4fea95acbf8a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "8db93cc2942c46168f4d716539920329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 8415294 live_row_count: 47 } } updated_tablets { tablet_id: "e9dace2ddc84456aa9f50378f3685951" state: RUNNING consensus_state { current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ece3e2ecf33a4c39bae992490a2f5cf0" state: RUNNING consensus_state { current_term: 2 leader_uuid: "8db93cc2942c46168f4d716539920329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 8415294 live_row_count: 10 } } sequence_number: 148
I20250115 02:52:49.905278 6792 raft_consensus.cc:1270] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 2 FOLLOWER]: Refusing update from remote peer 8db93cc2942c46168f4d716539920329: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250115 02:52:49.907670 6965 consensus_queue.cc:1035] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250115 02:52:49.966275 6871 raft_consensus.cc:1270] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Refusing update from remote peer 8db93cc2942c46168f4d716539920329: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250115 02:52:49.977540 6986 consensus_queue.cc:1035] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250115 02:52:50.200645 6634 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:50.751971 7011 raft_consensus.cc:491] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:50.752653 7011 raft_consensus.cc:513] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:50.759582 7011 leader_election.cc:290] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9e479ff36ff4e23a62d90bea40dbbf2 (127.2.36.126:46053), 7704565add4b41f08c2c7cd675fc98d8 (127.2.36.124:45585)
I20250115 02:52:50.760525 6459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e532bd0758d54ccca74891d5d555f832" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" is_pre_election: true
I20250115 02:52:50.762416 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e532bd0758d54ccca74891d5d555f832" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "7704565add4b41f08c2c7cd675fc98d8" is_pre_election: true
I20250115 02:52:50.763049 6589 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e532bd0758d54ccca74891d5d555f832 in term 2.
I20250115 02:52:50.766502 6500 leader_election.cc:304] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7704565add4b41f08c2c7cd675fc98d8, e532bd0758d54ccca74891d5d555f832; no voters:
I20250115 02:52:50.768064 7011 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250115 02:52:50.768424 7011 raft_consensus.cc:491] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:50.769064 7011 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:52:50.782615 7011 raft_consensus.cc:513] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:50.787022 7011 leader_election.cc:290] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [CANDIDATE]: Term 3 election: Requested vote from peers b9e479ff36ff4e23a62d90bea40dbbf2 (127.2.36.126:46053), 7704565add4b41f08c2c7cd675fc98d8 (127.2.36.124:45585)
I20250115 02:52:50.792115 6459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e532bd0758d54ccca74891d5d555f832" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2"
I20250115 02:52:50.793565 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e532bd0758d54ccca74891d5d555f832" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "7704565add4b41f08c2c7cd675fc98d8"
I20250115 02:52:50.794159 6589 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:52:50.818133 6589 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e532bd0758d54ccca74891d5d555f832 in term 3.
I20250115 02:52:50.837133 6500 leader_election.cc:304] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7704565add4b41f08c2c7cd675fc98d8, e532bd0758d54ccca74891d5d555f832; no voters: b9e479ff36ff4e23a62d90bea40dbbf2
I20250115 02:52:50.837911 7011 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 3 FOLLOWER]: Leader election won for term 3
I20250115 02:52:50.866552 7011 raft_consensus.cc:695] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 3 LEADER]: Becoming Leader. State: Replica: e532bd0758d54ccca74891d5d555f832, State: Running, Role: LEADER
I20250115 02:52:50.874831 7011 consensus_queue.cc:237] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:50.938722 7012 sys_catalog.cc:455] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e532bd0758d54ccca74891d5d555f832. Latest consensus state: current_term: 3 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:50.939589 7012 sys_catalog.cc:458] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:50.965898 7015 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:52:51.021315 7016 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e532bd0758d54ccca74891d5d555f832)
I20250115 02:52:51.023563 7016 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } }
I20250115 02:52:51.036639 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "7704565add4b41f08c2c7cd675fc98d8" is_pre_election: true
I20250115 02:52:51.037665 6589 raft_consensus.cc:2363] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9e479ff36ff4e23a62d90bea40dbbf2 for earlier term 2. Current term is 3.
I20250115 02:52:51.038822 6435 leader_election.cc:400] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7704565add4b41f08c2c7cd675fc98d8 (127.2.36.124:45585) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9e479ff36ff4e23a62d90bea40dbbf2 for earlier term 2. Current term is 3.
I20250115 02:52:51.039167 6435 leader_election.cc:403] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250115 02:52:51.047516 7017 raft_consensus.cc:491] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:51.036350 7016 leader_election.cc:290] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e532bd0758d54ccca74891d5d555f832 (127.2.36.125:39803), 7704565add4b41f08c2c7cd675fc98d8 (127.2.36.124:45585)
I20250115 02:52:51.055979 7016 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 1 FOLLOWER]: Advancing to term 3
I20250115 02:52:51.066402 6965 raft_consensus.cc:491] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:51.067029 6965 raft_consensus.cc:513] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:51.068954 6965 leader_election.cc:290] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ee170743349481cba07687363961f45 (127.2.36.66:35963), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:51.070501 6870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4" is_pre_election: true
I20250115 02:52:51.074797 7019 raft_consensus.cc:491] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:51.075179 7015 catalog_manager.cc:670] Loaded metadata for table ksck-test-table [id=653e85ea80884759bfa6d984105e31ac]
I20250115 02:52:51.048043 7017 raft_consensus.cc:513] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:51.077369 6870 raft_consensus.cc:2463] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 2.
I20250115 02:52:51.080485 6668 leader_election.cc:304] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8db93cc2942c46168f4d716539920329, a38e399727bc4503846b86b6aee45fa4; no voters:
I20250115 02:52:51.081399 7002 raft_consensus.cc:2798] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250115 02:52:51.082255 7017 leader_election.cc:290] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), 1ee170743349481cba07687363961f45 (127.2.36.66:35963)
I20250115 02:52:51.075312 7019 raft_consensus.cc:513] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:51.083266 6793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "a38e399727bc4503846b86b6aee45fa4" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45" is_pre_election: true
I20250115 02:52:51.083884 6793 raft_consensus.cc:2463] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a38e399727bc4503846b86b6aee45fa4 in term 2.
I20250115 02:52:51.084759 7019 leader_election.cc:290] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:51.085582 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "1ee170743349481cba07687363961f45" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8db93cc2942c46168f4d716539920329" is_pre_election: true
I20250115 02:52:51.085951 6835 leader_election.cc:304] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee170743349481cba07687363961f45, a38e399727bc4503846b86b6aee45fa4; no voters:
I20250115 02:52:51.081775 7002 raft_consensus.cc:491] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:51.087941 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "a38e399727bc4503846b86b6aee45fa4" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8db93cc2942c46168f4d716539920329" is_pre_election: true
I20250115 02:52:51.089253 7002 raft_consensus.cc:3054] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:52:51.087060 7016 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 7704565add4b41f08c2c7cd675fc98d8 (127.2.36.124:45585) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9e479ff36ff4e23a62d90bea40dbbf2 for earlier term 2. Current term is 3.
I20250115 02:52:51.090942 7017 raft_consensus.cc:2798] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250115 02:52:51.091325 7017 raft_consensus.cc:491] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:51.087270 6870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "1ee170743349481cba07687363961f45" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4" is_pre_election: true
I20250115 02:52:51.091652 7017 raft_consensus.cc:3054] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:52:51.109289 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45" is_pre_election: true
I20250115 02:52:51.110239 6791 raft_consensus.cc:2463] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 2.
I20250115 02:52:51.112366 7002 raft_consensus.cc:513] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:51.115916 6704 raft_consensus.cc:2388] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ee170743349481cba07687363961f45 in current term 3: Already voted for candidate 8db93cc2942c46168f4d716539920329 in this term.
I20250115 02:52:51.135200 6791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45"
I20250115 02:52:51.136090 6791 raft_consensus.cc:3054] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:52:51.142138 6871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "8db93cc2942c46168f4d716539920329" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a38e399727bc4503846b86b6aee45fa4"
I20250115 02:52:51.144873 7002 leader_election.cc:290] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 3 election: Requested vote from peers 1ee170743349481cba07687363961f45 (127.2.36.66:35963), a38e399727bc4503846b86b6aee45fa4 (127.2.36.67:37441)
I20250115 02:52:51.172407 7015 tablet_loader.cc:96] loaded metadata for tablet dca89dd97d744fdb996f4fea95acbf8a (table ksck-test-table [id=653e85ea80884759bfa6d984105e31ac])
I20250115 02:52:51.174113 7015 tablet_loader.cc:96] loaded metadata for tablet e9dace2ddc84456aa9f50378f3685951 (table ksck-test-table [id=653e85ea80884759bfa6d984105e31ac])
I20250115 02:52:51.179834 7015 tablet_loader.cc:96] loaded metadata for tablet ece3e2ecf33a4c39bae992490a2f5cf0 (table ksck-test-table [id=653e85ea80884759bfa6d984105e31ac])
I20250115 02:52:51.160156 7017 raft_consensus.cc:513] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:51.181677 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e532bd0758d54ccca74891d5d555f832" is_pre_election: true
I20250115 02:52:51.186318 6870 raft_consensus.cc:2388] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ee170743349481cba07687363961f45 in current term 3: Already voted for candidate a38e399727bc4503846b86b6aee45fa4 in this term.
I20250115 02:52:51.189348 6756 leader_election.cc:304] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ee170743349481cba07687363961f45; no voters: 8db93cc2942c46168f4d716539920329, a38e399727bc4503846b86b6aee45fa4
I20250115 02:52:51.181609 7015 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:52:51.199014 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "a38e399727bc4503846b86b6aee45fa4" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8db93cc2942c46168f4d716539920329"
I20250115 02:52:51.199892 6704 raft_consensus.cc:2388] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a38e399727bc4503846b86b6aee45fa4 in current term 3: Already voted for candidate 8db93cc2942c46168f4d716539920329 in this term.
I20250115 02:52:51.199072 6792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9dace2ddc84456aa9f50378f3685951" candidate_uuid: "a38e399727bc4503846b86b6aee45fa4" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1ee170743349481cba07687363961f45"
I20250115 02:52:51.207819 6834 leader_election.cc:304] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a38e399727bc4503846b86b6aee45fa4; no voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329
I20250115 02:52:51.199137 7017 leader_election.cc:290] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 3 election: Requested vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), 1ee170743349481cba07687363961f45 (127.2.36.66:35963)
I20250115 02:52:51.212076 7017 raft_consensus.cc:2743] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250115 02:52:51.201946 7015 catalog_manager.cc:1260] Loaded cluster ID: 7a5854e1cee94722828261428e087b51
I20250115 02:52:51.214154 7015 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:52:51.223665 6791 raft_consensus.cc:2463] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db93cc2942c46168f4d716539920329 in term 3.
I20250115 02:52:51.228495 7019 raft_consensus.cc:2743] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250115 02:52:51.239956 7015 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:52:51.244973 7015 catalog_manager.cc:5910] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832: Loaded TSK: 0
I20250115 02:52:51.246379 7015 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:52:51.283317 6668 leader_election.cc:304] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329; no voters: a38e399727bc4503846b86b6aee45fa4
I20250115 02:52:51.284693 6986 raft_consensus.cc:2798] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 3 FOLLOWER]: Leader election won for term 3
I20250115 02:52:51.285228 6986 raft_consensus.cc:695] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 3 LEADER]: Becoming Leader. State: Replica: 8db93cc2942c46168f4d716539920329, State: Running, Role: LEADER
I20250115 02:52:51.286069 6986 consensus_queue.cc:237] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:51.288244 6819 heartbeater.cc:501] Master 127.2.36.125:39803 was elected leader, sending a full tablet report...
I20250115 02:52:51.292696 6459 raft_consensus.cc:1270] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 3 FOLLOWER]: Refusing update from remote peer e532bd0758d54ccca74891d5d555f832: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20250115 02:52:51.294413 7012 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250115 02:52:51.296711 6897 heartbeater.cc:501] Master 127.2.36.125:39803 was elected leader, sending a full tablet report...
I20250115 02:52:51.298779 6730 heartbeater.cc:501] Master 127.2.36.125:39803 was elected leader, sending a full tablet report...
I20250115 02:52:51.305886 6512 catalog_manager.cc:5526] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 reported cstate change: term changed from 1 to 2, leader changed from 1ee170743349481cba07687363961f45 (127.2.36.66) to 8db93cc2942c46168f4d716539920329 (127.2.36.65). New cstate: current_term: 2 leader_uuid: "8db93cc2942c46168f4d716539920329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } }
I20250115 02:52:51.345247 7011 sys_catalog.cc:455] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:51.346097 7011 sys_catalog.cc:458] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:51.341245 7016 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e532bd0758d54ccca74891d5d555f832. Latest consensus state: current_term: 3 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:51.348295 6589 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 3 FOLLOWER]: Refusing update from remote peer e532bd0758d54ccca74891d5d555f832: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20250115 02:52:51.351732 7012 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250115 02:52:51.355612 7016 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:51.396394 7016 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:51.397194 7016 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:51.407533 6511 catalog_manager.cc:5526] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8db93cc2942c46168f4d716539920329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:51.453889 7032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:51.455097 7032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:51.455860 7011 sys_catalog.cc:455] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:51.456616 7011 sys_catalog.cc:458] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:51.466634 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e532bd0758d54ccca74891d5d555f832. Latest consensus state: current_term: 3 leader_uuid: "e532bd0758d54ccca74891d5d555f832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e479ff36ff4e23a62d90bea40dbbf2" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46053 } } peers { permanent_uuid: "e532bd0758d54ccca74891d5d555f832" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39803 } } peers { permanent_uuid: "7704565add4b41f08c2c7cd675fc98d8" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45585 } } }
I20250115 02:52:51.467433 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [sys.catalog]: This master's current role is: FOLLOWER
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/ksck_remote-test.cc:492: Failure
Failed
Bad status: Corruption: there are master consensus conflicts
I20250115 02:52:51.620131 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:52:51.690459 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:51.692029 2193 tablet_replica.cc:331] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329: stopping tablet replica
I20250115 02:52:51.692808 2193 raft_consensus.cc:2238] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 2 LEADER]: Raft consensus shutting down.
I20250115 02:52:51.693754 2193 raft_consensus.cc:2267] T ece3e2ecf33a4c39bae992490a2f5cf0 P 8db93cc2942c46168f4d716539920329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:51.696542 2193 tablet_replica.cc:331] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329: stopping tablet replica
I20250115 02:52:51.697312 2193 raft_consensus.cc:2238] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 3 LEADER]: Raft consensus shutting down.
I20250115 02:52:51.698112 2193 pending_rounds.cc:62] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329: Trying to abort 1 pending ops.
I20250115 02:52:51.698370 2193 pending_rounds.cc:69] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329: Aborting op as it isn't in flight: id { term: 3 index: 12 } timestamp: 7114381604007497728 op_type: NO_OP noop_request { }
I20250115 02:52:51.698832 2193 raft_consensus.cc:2883] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 02:52:51.699195 2193 raft_consensus.cc:2267] T e9dace2ddc84456aa9f50378f3685951 P 8db93cc2942c46168f4d716539920329 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:51.703096 2193 tablet_replica.cc:331] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329: stopping tablet replica
I20250115 02:52:51.703754 2193 raft_consensus.cc:2238] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:51.704838 2193 raft_consensus.cc:2267] T dca89dd97d744fdb996f4fea95acbf8a P 8db93cc2942c46168f4d716539920329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:51.735618 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:52:51.763432 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:52:51.849218 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:51.849951 2193 tablet_replica.cc:331] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45: stopping tablet replica
I20250115 02:52:51.850909 2193 raft_consensus.cc:2238] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:51.851485 2193 raft_consensus.cc:2267] T dca89dd97d744fdb996f4fea95acbf8a P 1ee170743349481cba07687363961f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:51.855376 2193 tablet_replica.cc:331] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45: stopping tablet replica
I20250115 02:52:51.855976 2193 raft_consensus.cc:2238] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:51.856436 2193 raft_consensus.cc:2267] T e9dace2ddc84456aa9f50378f3685951 P 1ee170743349481cba07687363961f45 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:51.858326 2193 tablet_replica.cc:331] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45: stopping tablet replica
I20250115 02:52:51.858774 2193 raft_consensus.cc:2238] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:51.859191 2193 raft_consensus.cc:2267] T ece3e2ecf33a4c39bae992490a2f5cf0 P 1ee170743349481cba07687363961f45 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:52.875463 7035 raft_consensus.cc:491] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8db93cc2942c46168f4d716539920329)
I20250115 02:52:52.876694 7035 raft_consensus.cc:513] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:52.878880 7035 leader_election.cc:290] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ee170743349481cba07687363961f45 (127.2.36.66:35963), 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663)
W20250115 02:52:52.881516 2193 thread.cc:535] Waited for 1000ms trying to join with diag-logger (tid 6814)
W20250115 02:52:52.887419 6835 leader_election.cc:336] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ee170743349481cba07687363961f45 (127.2.36.66:35963): Network error: Client connection negotiation failed: client connection to 127.2.36.66:35963: connect: Connection refused (error 111)
W20250115 02:52:52.888041 6834 leader_election.cc:336] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663): Network error: Client connection negotiation failed: client connection to 127.2.36.65:44663: connect: Connection refused (error 111)
I20250115 02:52:52.888414 6834 leader_election.cc:304] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a38e399727bc4503846b86b6aee45fa4; no voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329
I20250115 02:52:52.889074 7035 raft_consensus.cc:2743] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250115 02:52:53.127342 7035 raft_consensus.cc:491] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8db93cc2942c46168f4d716539920329)
I20250115 02:52:53.127892 7035 raft_consensus.cc:513] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:53.129843 7035 leader_election.cc:290] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), 1ee170743349481cba07687363961f45 (127.2.36.66:35963)
W20250115 02:52:53.133529 6834 leader_election.cc:336] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663): Network error: Client connection negotiation failed: client connection to 127.2.36.65:44663: connect: Connection refused (error 111)
W20250115 02:52:53.134191 6835 leader_election.cc:336] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ee170743349481cba07687363961f45 (127.2.36.66:35963): Network error: Client connection negotiation failed: client connection to 127.2.36.66:35963: connect: Connection refused (error 111)
I20250115 02:52:53.134548 6835 leader_election.cc:304] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a38e399727bc4503846b86b6aee45fa4; no voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329
I20250115 02:52:53.135233 7035 raft_consensus.cc:2743] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 02:52:53.475450 7035 raft_consensus.cc:491] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:53.475972 7035 raft_consensus.cc:513] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:53.478183 7035 leader_election.cc:290] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), 1ee170743349481cba07687363961f45 (127.2.36.66:35963)
W20250115 02:52:53.482715 6834 leader_election.cc:336] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663): Network error: Client connection negotiation failed: client connection to 127.2.36.65:44663: connect: Connection refused (error 111)
W20250115 02:52:53.483441 6835 leader_election.cc:336] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1ee170743349481cba07687363961f45 (127.2.36.66:35963): Network error: Client connection negotiation failed: client connection to 127.2.36.66:35963: connect: Connection refused (error 111)
I20250115 02:52:53.483918 6835 leader_election.cc:304] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a38e399727bc4503846b86b6aee45fa4; no voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329
I20250115 02:52:53.484653 7035 raft_consensus.cc:2743] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250115 02:52:53.882004 2193 thread.cc:535] Waited for 2000ms trying to join with diag-logger (tid 6814)
I20250115 02:52:54.702593 7040 raft_consensus.cc:491] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8db93cc2942c46168f4d716539920329)
I20250115 02:52:54.703290 7040 raft_consensus.cc:513] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } } peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } }
I20250115 02:52:54.705081 7040 leader_election.cc:290] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ee170743349481cba07687363961f45 (127.2.36.66:35963), 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663)
W20250115 02:52:54.707499 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.36.66:35963: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20250115 02:52:54.711787 6834 leader_election.cc:336] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663): Network error: Client connection negotiation failed: client connection to 127.2.36.65:44663: connect: Connection refused (error 111)
W20250115 02:52:54.712738 6835 leader_election.cc:336] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ee170743349481cba07687363961f45 (127.2.36.66:35963): Network error: Client connection negotiation failed: client connection to 127.2.36.66:35963: connect: Connection refused (error 111)
I20250115 02:52:54.713202 6835 leader_election.cc:304] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a38e399727bc4503846b86b6aee45fa4; no voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329
I20250115 02:52:54.713959 7040 raft_consensus.cc:2743] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250115 02:52:54.844712 7040 raft_consensus.cc:491] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8db93cc2942c46168f4d716539920329)
I20250115 02:52:54.845219 7040 raft_consensus.cc:513] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db93cc2942c46168f4d716539920329" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 44663 } } peers { permanent_uuid: "1ee170743349481cba07687363961f45" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 35963 } } peers { permanent_uuid: "a38e399727bc4503846b86b6aee45fa4" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37441 } }
I20250115 02:52:54.847208 7040 leader_election.cc:290] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663), 1ee170743349481cba07687363961f45 (127.2.36.66:35963)
W20250115 02:52:54.851259 6834 leader_election.cc:336] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8db93cc2942c46168f4d716539920329 (127.2.36.65:44663): Network error: Client connection negotiation failed: client connection to 127.2.36.65:44663: connect: Connection refused (error 111)
W20250115 02:52:54.852567 6835 leader_election.cc:336] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ee170743349481cba07687363961f45 (127.2.36.66:35963): Network error: Client connection negotiation failed: client connection to 127.2.36.66:35963: connect: Connection refused (error 111)
I20250115 02:52:54.852994 6835 leader_election.cc:304] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a38e399727bc4503846b86b6aee45fa4; no voters: 1ee170743349481cba07687363961f45, 8db93cc2942c46168f4d716539920329
I20250115 02:52:54.853668 7040 raft_consensus.cc:2743] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250115 02:52:54.882431 2193 thread.cc:535] Waited for 3000ms trying to join with diag-logger (tid 6814)
I20250115 02:52:55.316825 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:52:55.330729 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:52:55.355091 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:55.355700 2193 tablet_replica.cc:331] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4: stopping tablet replica
I20250115 02:52:55.356225 2193 raft_consensus.cc:2238] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:55.356722 2193 raft_consensus.cc:2267] T dca89dd97d744fdb996f4fea95acbf8a P a38e399727bc4503846b86b6aee45fa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:55.358682 2193 tablet_replica.cc:331] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4: stopping tablet replica
I20250115 02:52:55.359213 2193 raft_consensus.cc:2238] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:55.359619 2193 raft_consensus.cc:2267] T e9dace2ddc84456aa9f50378f3685951 P a38e399727bc4503846b86b6aee45fa4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:55.361409 2193 tablet_replica.cc:331] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4: stopping tablet replica
I20250115 02:52:55.361835 2193 raft_consensus.cc:2238] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:55.362185 2193 raft_consensus.cc:2267] T ece3e2ecf33a4c39bae992490a2f5cf0 P a38e399727bc4503846b86b6aee45fa4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:55.381868 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:52:55.395107 2193 master.cc:537] Master@127.2.36.126:46053 shutting down...
I20250115 02:52:55.414614 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:55.415194 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:55.415510 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P b9e479ff36ff4e23a62d90bea40dbbf2: stopping tablet replica
I20250115 02:52:55.434079 2193 master.cc:559] Master@127.2.36.126:46053 shutdown complete.
I20250115 02:52:55.447091 2193 master.cc:537] Master@127.2.36.125:39803 shutting down...
I20250115 02:52:55.464671 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 3 LEADER]: Raft consensus shutting down.
I20250115 02:52:55.465481 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:55.465806 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P e532bd0758d54ccca74891d5d555f832: stopping tablet replica
I20250115 02:52:55.485869 2193 master.cc:559] Master@127.2.36.125:39803 shutdown complete.
I20250115 02:52:55.499588 2193 master.cc:537] Master@127.2.36.124:45585 shutting down...
I20250115 02:52:55.515395 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:55.515916 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:55.516219 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7704565add4b41f08c2c7cd675fc98d8: stopping tablet replica
I20250115 02:52:55.534387 2193 master.cc:559] Master@127.2.36.124:45585 shutdown complete.
I20250115 02:52:55.554657 2193 test_util.cc:181] -----------------------------------------------
I20250115 02:52:55.554895 2193 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp.1736909525896969-2193-0
[ FAILED ] RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp (12666 ms)
[ RUN ] RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM
I20250115 02:52:55.557485 2193 test_util.cc:274] Using random seed: -968748708
I20250115 02:52:55.558399 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:33081,127.2.36.125:38391,127.2.36.124:41773
I20250115 02:52:55.559732 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:55.565342 7044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:55.565941 7045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:55.566951 7047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:55.567732 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:55.568519 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:55.568678 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:55.568783 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909575568773 us; error 0 us; skew 500 ppm
I20250115 02:52:55.569228 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:55.575127 2193 webserver.cc:458] Webserver started at http://127.2.36.126:36079/ using document root <none> and password file <none>
I20250115 02:52:55.575546 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:55.575687 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:55.575896 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:55.576931 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "4c56a121def049488a306eace8ffebb6"
format_stamp: "Formatted at 2025-01-15 02:52:55 on dist-test-slave-1lv2"
I20250115 02:52:55.580960 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:52:55.583997 7052 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:55.584733 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:52:55.584986 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "4c56a121def049488a306eace8ffebb6"
format_stamp: "Formatted at 2025-01-15 02:52:55 on dist-test-slave-1lv2"
I20250115 02:52:55.585229 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:55.601693 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:55.602861 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:55.637607 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:33081
I20250115 02:52:55.637704 7103 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:33081 every 8 connection(s)
I20250115 02:52:55.641217 7104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:55.642061 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:55.648172 7106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:55.648418 7104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:55.649032 7107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:55.649616 7109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:55.651419 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:55.653206 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:55.653458 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:55.653625 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909575653612 us; error 0 us; skew 500 ppm
I20250115 02:52:55.654142 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:55.656865 2193 webserver.cc:458] Webserver started at http://127.2.36.125:40339/ using document root <none> and password file <none>
I20250115 02:52:55.657413 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:55.657630 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:55.657932 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:55.659262 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "11c08d58238c49449fcba36cf3f55fbb"
format_stamp: "Formatted at 2025-01-15 02:52:55 on dist-test-slave-1lv2"
I20250115 02:52:55.660600 7104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:55.664537 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
W20250115 02:52:55.665984 7104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:38391: Network error: Client connection negotiation failed: client connection to 127.2.36.125:38391: connect: Connection refused (error 111)
I20250115 02:52:55.668262 7117 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:55.668973 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250115 02:52:55.669237 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "11c08d58238c49449fcba36cf3f55fbb"
format_stamp: "Formatted at 2025-01-15 02:52:55 on dist-test-slave-1lv2"
I20250115 02:52:55.669498 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:55.683061 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:55.684154 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:55.693466 7104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } attempt: 1
W20250115 02:52:55.698096 7104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:38391: Network error: Client connection negotiation failed: client connection to 127.2.36.125:38391: connect: Connection refused (error 111)
I20250115 02:52:55.720247 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:38391
I20250115 02:52:55.720379 7168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:38391 every 8 connection(s)
I20250115 02:52:55.723928 7169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:55.723963 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:55.730450 7171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:55.731282 7172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:55.731670 7169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:55.732221 7174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:55.733973 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:55.735064 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:55.735272 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:55.735420 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909575735402 us; error 0 us; skew 500 ppm
I20250115 02:52:55.736091 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:55.738620 2193 webserver.cc:458] Webserver started at http://127.2.36.124:38479/ using document root <none> and password file <none>
I20250115 02:52:55.739282 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:55.739517 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:55.739823 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:55.741109 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "2595e4cdffa24599aa59225bb41bdb25"
format_stamp: "Formatted at 2025-01-15 02:52:55 on dist-test-slave-1lv2"
I20250115 02:52:55.743100 7169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:55.747406 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250115 02:52:55.751246 7181 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:55.752166 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250115 02:52:55.752548 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "2595e4cdffa24599aa59225bb41bdb25"
format_stamp: "Formatted at 2025-01-15 02:52:55 on dist-test-slave-1lv2"
I20250115 02:52:55.752827 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:55.753858 7169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:55.759357 7169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:41773: Network error: Client connection negotiation failed: client connection to 127.2.36.124:41773: connect: Connection refused (error 111)
I20250115 02:52:55.772588 7104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } attempt: 2
I20250115 02:52:55.781571 7104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:55.785239 7104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:41773: Network error: Client connection negotiation failed: client connection to 127.2.36.124:41773: connect: Connection refused (error 111)
I20250115 02:52:55.799700 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:55.800894 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:55.824836 7169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } attempt: 1
W20250115 02:52:55.829103 7169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:41773: Network error: Client connection negotiation failed: client connection to 127.2.36.124:41773: connect: Connection refused (error 111)
I20250115 02:52:55.830659 7104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } attempt: 1
W20250115 02:52:55.834745 7104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:41773: Network error: Client connection negotiation failed: client connection to 127.2.36.124:41773: connect: Connection refused (error 111)
I20250115 02:52:55.839138 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:41773
I20250115 02:52:55.839247 7233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:41773 every 8 connection(s)
I20250115 02:52:55.842016 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:52:55.842856 7234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:55.848552 7234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:55.858158 7234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:55.866602 7234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:55.880694 7234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25: Bootstrap starting.
I20250115 02:52:55.885141 7234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:55.889015 7234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25: No bootstrap required, opened a new log
I20250115 02:52:55.889277 7104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } attempt: 2
I20250115 02:52:55.891638 7234 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } }
I20250115 02:52:55.892297 7234 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:55.892586 7234 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2595e4cdffa24599aa59225bb41bdb25, State: Initialized, Role: FOLLOWER
I20250115 02:52:55.893225 7234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } }
I20250115 02:52:55.893535 7169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } attempt: 2
I20250115 02:52:55.895277 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } } }
I20250115 02:52:55.896059 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:55.898437 7234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:55.907794 7104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6: Bootstrap starting.
I20250115 02:52:55.910995 7169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb: Bootstrap starting.
I20250115 02:52:55.914477 7104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:55.916781 7169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:55.919296 7104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6: No bootstrap required, opened a new log
W20250115 02:52:55.920476 7251 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:55.920800 7251 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:55.921627 7169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb: No bootstrap required, opened a new log
I20250115 02:52:55.921568 7104 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } }
I20250115 02:52:55.922200 7104 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:55.922420 7104 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c56a121def049488a306eace8ffebb6, State: Initialized, Role: FOLLOWER
I20250115 02:52:55.922983 7104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } }
I20250115 02:52:55.923887 7169 raft_consensus.cc:357] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } }
I20250115 02:52:55.924623 7169 raft_consensus.cc:383] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:55.924930 7169 raft_consensus.cc:738] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c08d58238c49449fcba36cf3f55fbb, State: Initialized, Role: FOLLOWER
I20250115 02:52:55.924737 7254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } } }
I20250115 02:52:55.925565 7254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:55.925959 7104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:55.925638 7169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } }
I20250115 02:52:55.927722 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } } }
I20250115 02:52:55.928520 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:55.929236 7169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:55.938323 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20250115 02:52:55.941433 7271 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:55.941726 7271 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:55.942548 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:52:55.943939 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:55.945328 7277 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:55.945591 7277 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:52:55.950668 7278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:55.951292 7279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:55.952245 7281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:55.953320 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:55.954042 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:55.954252 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:55.954397 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909575954382 us; error 0 us; skew 500 ppm
I20250115 02:52:55.954952 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:55.957150 2193 webserver.cc:458] Webserver started at http://127.2.36.65:44503/ using document root <none> and password file <none>
I20250115 02:52:55.957559 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:55.957720 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:55.957939 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:55.958958 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "252fc48398ea43f683ffb1ccfd923837"
format_stamp: "Formatted at 2025-01-15 02:52:55 on dist-test-slave-1lv2"
I20250115 02:52:55.963021 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250115 02:52:55.965873 7286 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:55.966524 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:52:55.966820 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "252fc48398ea43f683ffb1ccfd923837"
format_stamp: "Formatted at 2025-01-15 02:52:55 on dist-test-slave-1lv2"
I20250115 02:52:55.967079 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:55.987182 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:55.988330 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:55.989691 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:55.991994 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:55.992197 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:55.992415 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:55.992564 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:56.031682 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:46765
I20250115 02:52:56.031778 7348 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:46765 every 8 connection(s)
I20250115 02:52:56.051843 7349 heartbeater.cc:346] Connected to a master server at 127.2.36.124:41773
I20250115 02:52:56.052276 7349 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:56.053125 7349 heartbeater.cc:509] Master 127.2.36.124:41773 requested a full tablet report, sending...
I20250115 02:52:56.056200 7199 ts_manager.cc:194] Registered new tserver with Master: 252fc48398ea43f683ffb1ccfd923837 (127.2.36.65:46765)
I20250115 02:52:56.059762 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:56.060297 7350 heartbeater.cc:346] Connected to a master server at 127.2.36.125:38391
I20250115 02:52:56.060703 7350 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:56.061545 7350 heartbeater.cc:509] Master 127.2.36.125:38391 requested a full tablet report, sending...
I20250115 02:52:56.063139 7351 heartbeater.cc:346] Connected to a master server at 127.2.36.126:33081
I20250115 02:52:56.063542 7351 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:56.064117 7134 ts_manager.cc:194] Registered new tserver with Master: 252fc48398ea43f683ffb1ccfd923837 (127.2.36.65:46765)
I20250115 02:52:56.064417 7351 heartbeater.cc:509] Master 127.2.36.126:33081 requested a full tablet report, sending...
I20250115 02:52:56.067155 7069 ts_manager.cc:194] Registered new tserver with Master: 252fc48398ea43f683ffb1ccfd923837 (127.2.36.65:46765)
W20250115 02:52:56.068821 7356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:56.071420 7357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:56.078079 2193 server_base.cc:1034] running on GCE node
W20250115 02:52:56.079145 7359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:56.079962 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:56.080202 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:56.080371 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909576080353 us; error 0 us; skew 500 ppm
I20250115 02:52:56.080868 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:56.083907 2193 webserver.cc:458] Webserver started at http://127.2.36.66:43235/ using document root <none> and password file <none>
I20250115 02:52:56.084434 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:56.084604 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:56.084827 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:56.085793 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "a530070f493646f99d788c6da6ea3e23"
format_stamp: "Formatted at 2025-01-15 02:52:56 on dist-test-slave-1lv2"
I20250115 02:52:56.090245 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:52:56.093719 7364 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:56.094658 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:52:56.094993 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "a530070f493646f99d788c6da6ea3e23"
format_stamp: "Formatted at 2025-01-15 02:52:56 on dist-test-slave-1lv2"
I20250115 02:52:56.095345 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:56.119290 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:56.120589 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:56.122311 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:56.124907 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:56.125104 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:56.125335 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:56.125480 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:56.171411 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:41537
I20250115 02:52:56.171510 7426 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:41537 every 8 connection(s)
I20250115 02:52:56.216919 7428 heartbeater.cc:346] Connected to a master server at 127.2.36.125:38391
I20250115 02:52:56.217418 7428 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:56.218314 7428 heartbeater.cc:509] Master 127.2.36.125:38391 requested a full tablet report, sending...
I20250115 02:52:56.218783 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:56.220460 7427 heartbeater.cc:346] Connected to a master server at 127.2.36.124:41773
I20250115 02:52:56.220808 7427 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:56.221580 7427 heartbeater.cc:509] Master 127.2.36.124:41773 requested a full tablet report, sending...
I20250115 02:52:56.221612 7134 ts_manager.cc:194] Registered new tserver with Master: a530070f493646f99d788c6da6ea3e23 (127.2.36.66:41537)
I20250115 02:52:56.223227 7429 heartbeater.cc:346] Connected to a master server at 127.2.36.126:33081
I20250115 02:52:56.224334 7199 ts_manager.cc:194] Registered new tserver with Master: a530070f493646f99d788c6da6ea3e23 (127.2.36.66:41537)
I20250115 02:52:56.225955 7429 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:56.226666 7429 heartbeater.cc:509] Master 127.2.36.126:33081 requested a full tablet report, sending...
I20250115 02:52:56.229434 7069 ts_manager.cc:194] Registered new tserver with Master: a530070f493646f99d788c6da6ea3e23 (127.2.36.66:41537)
W20250115 02:52:56.230746 7434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:56.231408 7435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:56.233371 2193 server_base.cc:1034] running on GCE node
W20250115 02:52:56.234723 7437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:56.235710 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:56.236035 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:56.236317 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909576236297 us; error 0 us; skew 500 ppm
I20250115 02:52:56.237066 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:56.237635 7239 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:56.238111 7239 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } }
I20250115 02:52:56.240226 7239 leader_election.cc:290] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c56a121def049488a306eace8ffebb6 (127.2.36.126:33081), 11c08d58238c49449fcba36cf3f55fbb (127.2.36.125:38391)
I20250115 02:52:56.241262 7079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2595e4cdffa24599aa59225bb41bdb25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c56a121def049488a306eace8ffebb6" is_pre_election: true
I20250115 02:52:56.241775 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2595e4cdffa24599aa59225bb41bdb25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c08d58238c49449fcba36cf3f55fbb" is_pre_election: true
I20250115 02:52:56.241940 7079 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2595e4cdffa24599aa59225bb41bdb25 in term 0.
I20250115 02:52:56.242455 7144 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2595e4cdffa24599aa59225bb41bdb25 in term 0.
I20250115 02:52:56.243484 7186 leader_election.cc:304] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2595e4cdffa24599aa59225bb41bdb25, 4c56a121def049488a306eace8ffebb6; no voters:
I20250115 02:52:56.243777 2193 webserver.cc:458] Webserver started at http://127.2.36.67:39713/ using document root <none> and password file <none>
I20250115 02:52:56.244423 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:56.244727 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:56.244962 7441 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:56.245100 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:56.245481 7441 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:56.246134 7441 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:56.246729 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "bcbc16aa7c0d44cf9eb8a24564749ece"
format_stamp: "Formatted at 2025-01-15 02:52:56 on dist-test-slave-1lv2"
I20250115 02:52:56.253782 7441 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } }
I20250115 02:52:56.255677 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20250115 02:52:56.258631 7079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2595e4cdffa24599aa59225bb41bdb25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c56a121def049488a306eace8ffebb6"
I20250115 02:52:56.259297 7079 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:56.259620 7441 leader_election.cc:290] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [CANDIDATE]: Term 1 election: Requested vote from peers 4c56a121def049488a306eace8ffebb6 (127.2.36.126:33081), 11c08d58238c49449fcba36cf3f55fbb (127.2.36.125:38391)
I20250115 02:52:56.260874 7443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:56.261447 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2595e4cdffa24599aa59225bb41bdb25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c08d58238c49449fcba36cf3f55fbb"
I20250115 02:52:56.261996 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250115 02:52:56.262082 7144 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:56.262470 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "bcbc16aa7c0d44cf9eb8a24564749ece"
format_stamp: "Formatted at 2025-01-15 02:52:56 on dist-test-slave-1lv2"
I20250115 02:52:56.262871 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:56.268432 7144 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2595e4cdffa24599aa59225bb41bdb25 in term 1.
I20250115 02:52:56.270643 7079 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2595e4cdffa24599aa59225bb41bdb25 in term 1.
I20250115 02:52:56.271301 7183 leader_election.cc:304] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c08d58238c49449fcba36cf3f55fbb, 2595e4cdffa24599aa59225bb41bdb25; no voters:
I20250115 02:52:56.272193 7441 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:56.272529 7441 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 1 LEADER]: Becoming Leader. State: Replica: 2595e4cdffa24599aa59225bb41bdb25, State: Running, Role: LEADER
I20250115 02:52:56.273247 7441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } }
I20250115 02:52:56.285802 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2595e4cdffa24599aa59225bb41bdb25. Latest consensus state: current_term: 1 leader_uuid: "2595e4cdffa24599aa59225bb41bdb25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } } }
I20250115 02:52:56.286571 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:56.292644 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:56.294023 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:56.296200 7449 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:52:56.298243 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:56.303853 7449 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:52:56.307049 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:56.307286 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:56.307526 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:56.307665 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:56.314194 7144 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [term 1 FOLLOWER]: Refusing update from remote peer 2595e4cdffa24599aa59225bb41bdb25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:56.315470 7079 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [term 1 FOLLOWER]: Refusing update from remote peer 2595e4cdffa24599aa59225bb41bdb25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:56.315593 7441 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:56.322335 7441 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:56.329613 7254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2595e4cdffa24599aa59225bb41bdb25. Latest consensus state: current_term: 1 leader_uuid: "2595e4cdffa24599aa59225bb41bdb25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } } }
I20250115 02:52:56.330364 7254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:56.341759 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2595e4cdffa24599aa59225bb41bdb25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } } }
I20250115 02:52:56.342608 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:56.343282 7449 catalog_manager.cc:1348] Generated new cluster ID: 05fb5033948f4e5491df2ddfa15418da
I20250115 02:52:56.343611 7449 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:52:56.346820 7254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2595e4cdffa24599aa59225bb41bdb25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } } }
I20250115 02:52:56.347558 7254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:56.356343 7441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2595e4cdffa24599aa59225bb41bdb25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } } }
I20250115 02:52:56.357193 7441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:56.358119 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2595e4cdffa24599aa59225bb41bdb25. Latest consensus state: current_term: 1 leader_uuid: "2595e4cdffa24599aa59225bb41bdb25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } } }
I20250115 02:52:56.358886 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:56.382198 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2595e4cdffa24599aa59225bb41bdb25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c56a121def049488a306eace8ffebb6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33081 } } peers { permanent_uuid: "11c08d58238c49449fcba36cf3f55fbb" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38391 } } peers { permanent_uuid: "2595e4cdffa24599aa59225bb41bdb25" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41773 } } }
I20250115 02:52:56.382988 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:56.390110 7449 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:52:56.391855 7449 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:52:56.436628 7449 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25: Generated new TSK 0
I20250115 02:52:56.437378 7449 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:52:56.449986 7199 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46404
I20250115 02:52:56.460435 7199 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46418
I20250115 02:52:56.464888 7349 heartbeater.cc:501] Master 127.2.36.124:41773 was elected leader, sending a full tablet report...
I20250115 02:52:56.473001 7427 heartbeater.cc:501] Master 127.2.36.124:41773 was elected leader, sending a full tablet report...
I20250115 02:52:56.495046 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:43115
I20250115 02:52:56.495146 7517 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:43115 every 8 connection(s)
I20250115 02:52:56.524437 7519 heartbeater.cc:346] Connected to a master server at 127.2.36.125:38391
I20250115 02:52:56.524875 7519 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:56.525744 7519 heartbeater.cc:509] Master 127.2.36.125:38391 requested a full tablet report, sending...
I20250115 02:52:56.527985 7518 heartbeater.cc:346] Connected to a master server at 127.2.36.124:41773
I20250115 02:52:56.528296 7518 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:56.528623 7133 ts_manager.cc:194] Registered new tserver with Master: bcbc16aa7c0d44cf9eb8a24564749ece (127.2.36.67:43115)
I20250115 02:52:56.528987 7518 heartbeater.cc:509] Master 127.2.36.124:41773 requested a full tablet report, sending...
I20250115 02:52:56.530947 7199 ts_manager.cc:194] Registered new tserver with Master: bcbc16aa7c0d44cf9eb8a24564749ece (127.2.36.67:43115)
I20250115 02:52:56.532677 7199 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46420
I20250115 02:52:56.540664 7520 heartbeater.cc:346] Connected to a master server at 127.2.36.126:33081
I20250115 02:52:56.541080 7520 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:56.541663 7520 heartbeater.cc:509] Master 127.2.36.126:33081 requested a full tablet report, sending...
I20250115 02:52:56.543778 7069 ts_manager.cc:194] Registered new tserver with Master: bcbc16aa7c0d44cf9eb8a24564749ece (127.2.36.67:43115)
I20250115 02:52:56.544014 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.043106255s
I20250115 02:52:56.546638 7518 heartbeater.cc:501] Master 127.2.36.124:41773 was elected leader, sending a full tablet report...
I20250115 02:52:56.595876 7199 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46432:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:52:56.598794 7199 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:52:56.708175 7482 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8b6c42fc174d5382f4cf9d0228dd33 (DEFAULT_TABLE table=ksck-test-table [id=ae82358672e946bc93c7870ce218ea3d]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:56.708624 7483 tablet_service.cc:1467] Processing CreateTablet for tablet edcebc6c1cec4f1792bdd241fbc68a36 (DEFAULT_TABLE table=ksck-test-table [id=ae82358672e946bc93c7870ce218ea3d]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:56.709623 7482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f8b6c42fc174d5382f4cf9d0228dd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:56.711778 7481 tablet_service.cc:1467] Processing CreateTablet for tablet b487d84cdd4a49d78c81a671b8027339 (DEFAULT_TABLE table=ksck-test-table [id=ae82358672e946bc93c7870ce218ea3d]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:56.713088 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b487d84cdd4a49d78c81a671b8027339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:56.714789 7483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edcebc6c1cec4f1792bdd241fbc68a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:56.739713 7313 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8b6c42fc174d5382f4cf9d0228dd33 (DEFAULT_TABLE table=ksck-test-table [id=ae82358672e946bc93c7870ce218ea3d]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:56.739915 7314 tablet_service.cc:1467] Processing CreateTablet for tablet edcebc6c1cec4f1792bdd241fbc68a36 (DEFAULT_TABLE table=ksck-test-table [id=ae82358672e946bc93c7870ce218ea3d]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:56.741097 7313 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f8b6c42fc174d5382f4cf9d0228dd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:56.743628 7312 tablet_service.cc:1467] Processing CreateTablet for tablet b487d84cdd4a49d78c81a671b8027339 (DEFAULT_TABLE table=ksck-test-table [id=ae82358672e946bc93c7870ce218ea3d]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:56.748410 7312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b487d84cdd4a49d78c81a671b8027339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:56.749701 7392 tablet_service.cc:1467] Processing CreateTablet for tablet edcebc6c1cec4f1792bdd241fbc68a36 (DEFAULT_TABLE table=ksck-test-table [id=ae82358672e946bc93c7870ce218ea3d]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:56.750994 7392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edcebc6c1cec4f1792bdd241fbc68a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:56.751364 7314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edcebc6c1cec4f1792bdd241fbc68a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:56.753429 7391 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8b6c42fc174d5382f4cf9d0228dd33 (DEFAULT_TABLE table=ksck-test-table [id=ae82358672e946bc93c7870ce218ea3d]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:56.754720 7391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f8b6c42fc174d5382f4cf9d0228dd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:56.754863 7537 tablet_bootstrap.cc:492] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece: Bootstrap starting.
I20250115 02:52:56.755893 7390 tablet_service.cc:1467] Processing CreateTablet for tablet b487d84cdd4a49d78c81a671b8027339 (DEFAULT_TABLE table=ksck-test-table [id=ae82358672e946bc93c7870ce218ea3d]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:56.760491 7537 tablet_bootstrap.cc:654] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:56.765458 7390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b487d84cdd4a49d78c81a671b8027339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:56.838882 7539 tablet_bootstrap.cc:492] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23: Bootstrap starting.
I20250115 02:52:56.842746 7540 tablet_bootstrap.cc:492] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837: Bootstrap starting.
I20250115 02:52:56.845984 7537 tablet_bootstrap.cc:492] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece: No bootstrap required, opened a new log
I20250115 02:52:56.846419 7537 ts_tablet_manager.cc:1397] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece: Time spent bootstrapping tablet: real 0.092s user 0.013s sys 0.022s
I20250115 02:52:56.851406 7540 tablet_bootstrap.cc:654] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:56.854038 7539 tablet_bootstrap.cc:654] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:56.859434 7537 raft_consensus.cc:357] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.860270 7537 raft_consensus.cc:383] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:56.860605 7537 raft_consensus.cc:738] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbc16aa7c0d44cf9eb8a24564749ece, State: Initialized, Role: FOLLOWER
I20250115 02:52:56.861348 7537 consensus_queue.cc:260] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.864923 7540 tablet_bootstrap.cc:492] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837: No bootstrap required, opened a new log
I20250115 02:52:56.865479 7540 ts_tablet_manager.cc:1397] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.009s
I20250115 02:52:56.868336 7540 raft_consensus.cc:357] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.869864 7540 raft_consensus.cc:383] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:56.870201 7540 raft_consensus.cc:738] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252fc48398ea43f683ffb1ccfd923837, State: Initialized, Role: FOLLOWER
I20250115 02:52:56.871068 7540 consensus_queue.cc:260] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.881033 7540 ts_tablet_manager.cc:1428] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837: Time spent starting tablet: real 0.015s user 0.006s sys 0.006s
I20250115 02:52:56.882555 7537 ts_tablet_manager.cc:1428] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece: Time spent starting tablet: real 0.036s user 0.009s sys 0.017s
I20250115 02:52:56.883730 7537 tablet_bootstrap.cc:492] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece: Bootstrap starting.
I20250115 02:52:56.885877 7540 tablet_bootstrap.cc:492] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837: Bootstrap starting.
I20250115 02:52:56.890125 7537 tablet_bootstrap.cc:654] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:56.893457 7539 tablet_bootstrap.cc:492] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23: No bootstrap required, opened a new log
I20250115 02:52:56.894018 7539 ts_tablet_manager.cc:1397] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.011s
I20250115 02:52:56.901520 7537 tablet_bootstrap.cc:492] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece: No bootstrap required, opened a new log
I20250115 02:52:56.901929 7540 tablet_bootstrap.cc:654] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:56.902199 7537 ts_tablet_manager.cc:1397] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.011s
I20250115 02:52:56.902935 7539 raft_consensus.cc:357] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.905417 7546 raft_consensus.cc:491] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:56.905169 7537 raft_consensus.cc:357] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.905309 7539 raft_consensus.cc:383] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:56.906071 7537 raft_consensus.cc:383] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:56.905889 7546 raft_consensus.cc:513] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.906461 7539 raft_consensus.cc:738] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a530070f493646f99d788c6da6ea3e23, State: Initialized, Role: FOLLOWER
I20250115 02:52:56.907366 7539 consensus_queue.cc:260] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.910566 7540 tablet_bootstrap.cc:492] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837: No bootstrap required, opened a new log
I20250115 02:52:56.911126 7540 ts_tablet_manager.cc:1397] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.004s
I20250115 02:52:56.911909 7546 leader_election.cc:290] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 252fc48398ea43f683ffb1ccfd923837 (127.2.36.65:46765), a530070f493646f99d788c6da6ea3e23 (127.2.36.66:41537)
I20250115 02:52:56.912474 7537 raft_consensus.cc:738] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbc16aa7c0d44cf9eb8a24564749ece, State: Initialized, Role: FOLLOWER
I20250115 02:52:56.913304 7537 consensus_queue.cc:260] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.913861 7540 raft_consensus.cc:357] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } } peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } }
I20250115 02:52:56.914644 7540 raft_consensus.cc:383] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:56.914964 7540 raft_consensus.cc:738] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252fc48398ea43f683ffb1ccfd923837, State: Initialized, Role: FOLLOWER
I20250115 02:52:56.915652 7540 consensus_queue.cc:260] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } } peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } }
I20250115 02:52:56.916426 7537 ts_tablet_manager.cc:1428] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20250115 02:52:56.917450 7537 tablet_bootstrap.cc:492] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece: Bootstrap starting.
I20250115 02:52:56.917829 7540 ts_tablet_manager.cc:1428] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20250115 02:52:56.923765 7537 tablet_bootstrap.cc:654] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:56.925455 7540 tablet_bootstrap.cc:492] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837: Bootstrap starting.
I20250115 02:52:56.930676 7324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b487d84cdd4a49d78c81a671b8027339" candidate_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252fc48398ea43f683ffb1ccfd923837" is_pre_election: true
I20250115 02:52:56.931499 7324 raft_consensus.cc:2463] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcbc16aa7c0d44cf9eb8a24564749ece in term 0.
I20250115 02:52:56.932976 7445 leader_election.cc:304] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252fc48398ea43f683ffb1ccfd923837, bcbc16aa7c0d44cf9eb8a24564749ece; no voters:
I20250115 02:52:56.933880 7546 raft_consensus.cc:2798] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:56.934243 7546 raft_consensus.cc:491] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:56.934571 7546 raft_consensus.cc:3054] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:56.935837 7540 tablet_bootstrap.cc:654] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:56.941731 7546 raft_consensus.cc:513] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.949141 7537 tablet_bootstrap.cc:492] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece: No bootstrap required, opened a new log
I20250115 02:52:56.949739 7537 ts_tablet_manager.cc:1397] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.009s
I20250115 02:52:56.950918 7277 catalog_manager.cc:1260] Loaded cluster ID: 05fb5033948f4e5491df2ddfa15418da
I20250115 02:52:56.951277 7277 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb: loading cluster ID for follower catalog manager: success
I20250115 02:52:56.952862 7537 raft_consensus.cc:357] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } } peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } }
I20250115 02:52:56.954109 7537 raft_consensus.cc:383] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:56.954582 7537 raft_consensus.cc:738] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbc16aa7c0d44cf9eb8a24564749ece, State: Initialized, Role: FOLLOWER
I20250115 02:52:56.953704 7546 leader_election.cc:290] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [CANDIDATE]: Term 1 election: Requested vote from peers 252fc48398ea43f683ffb1ccfd923837 (127.2.36.65:46765), a530070f493646f99d788c6da6ea3e23 (127.2.36.66:41537)
I20250115 02:52:56.955328 7271 catalog_manager.cc:1260] Loaded cluster ID: 05fb5033948f4e5491df2ddfa15418da
I20250115 02:52:56.955706 7537 consensus_queue.cc:260] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } } peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } }
I20250115 02:52:56.956975 7324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b487d84cdd4a49d78c81a671b8027339" candidate_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252fc48398ea43f683ffb1ccfd923837"
I20250115 02:52:56.957578 7324 raft_consensus.cc:3054] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:56.957995 7537 ts_tablet_manager.cc:1428] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250115 02:52:56.958786 7277 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb: acquiring CA information for follower catalog manager: success
I20250115 02:52:56.961681 7540 tablet_bootstrap.cc:492] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837: No bootstrap required, opened a new log
I20250115 02:52:56.962244 7540 ts_tablet_manager.cc:1397] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837: Time spent bootstrapping tablet: real 0.044s user 0.012s sys 0.000s
I20250115 02:52:56.962706 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b487d84cdd4a49d78c81a671b8027339" candidate_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a530070f493646f99d788c6da6ea3e23"
I20250115 02:52:56.963339 7401 raft_consensus.cc:3054] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:56.965397 7540 raft_consensus.cc:357] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.966153 7540 raft_consensus.cc:383] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:56.966435 7540 raft_consensus.cc:738] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252fc48398ea43f683ffb1ccfd923837, State: Initialized, Role: FOLLOWER
I20250115 02:52:56.967195 7540 consensus_queue.cc:260] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.962731 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b487d84cdd4a49d78c81a671b8027339" candidate_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a530070f493646f99d788c6da6ea3e23" is_pre_election: true
I20250115 02:52:56.970096 7540 ts_tablet_manager.cc:1428] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250115 02:52:56.972467 7401 raft_consensus.cc:2463] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcbc16aa7c0d44cf9eb8a24564749ece in term 1.
I20250115 02:52:56.973706 7445 leader_election.cc:304] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a530070f493646f99d788c6da6ea3e23, bcbc16aa7c0d44cf9eb8a24564749ece; no voters:
I20250115 02:52:56.974743 7543 raft_consensus.cc:2798] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:56.975247 7543 raft_consensus.cc:695] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 LEADER]: Becoming Leader. State: Replica: bcbc16aa7c0d44cf9eb8a24564749ece, State: Running, Role: LEADER
I20250115 02:52:56.976029 7543 consensus_queue.cc:237] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:56.982573 7277 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:56.955664 7271 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6: loading cluster ID for follower catalog manager: success
I20250115 02:52:56.991345 7324 raft_consensus.cc:2463] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcbc16aa7c0d44cf9eb8a24564749ece in term 1.
I20250115 02:52:56.994611 7271 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6: acquiring CA information for follower catalog manager: success
I20250115 02:52:57.000238 7543 raft_consensus.cc:491] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:57.000833 7543 raft_consensus.cc:513] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:57.003922 7539 ts_tablet_manager.cc:1428] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23: Time spent starting tablet: real 0.103s user 0.023s sys 0.019s
I20250115 02:52:57.004789 7324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8b6c42fc174d5382f4cf9d0228dd33" candidate_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252fc48398ea43f683ffb1ccfd923837" is_pre_election: true
I20250115 02:52:57.005561 7324 raft_consensus.cc:2463] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcbc16aa7c0d44cf9eb8a24564749ece in term 0.
I20250115 02:52:57.006681 7445 leader_election.cc:304] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252fc48398ea43f683ffb1ccfd923837, bcbc16aa7c0d44cf9eb8a24564749ece; no voters:
I20250115 02:52:57.007560 7546 raft_consensus.cc:2798] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:57.007908 7546 raft_consensus.cc:491] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:57.008247 7546 raft_consensus.cc:3054] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:57.010228 7543 leader_election.cc:290] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 252fc48398ea43f683ffb1ccfd923837 (127.2.36.65:46765), a530070f493646f99d788c6da6ea3e23 (127.2.36.66:41537)
I20250115 02:52:57.011660 7198 catalog_manager.cc:5526] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece reported cstate change: term changed from 0 to 1, leader changed from <none> to bcbc16aa7c0d44cf9eb8a24564749ece (127.2.36.67). New cstate: current_term: 1 leader_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } health_report { overall_health: HEALTHY } } }
I20250115 02:52:57.016247 7271 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:57.018826 7539 tablet_bootstrap.cc:492] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23: Bootstrap starting.
I20250115 02:52:57.021286 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8b6c42fc174d5382f4cf9d0228dd33" candidate_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a530070f493646f99d788c6da6ea3e23" is_pre_election: true
W20250115 02:52:57.022581 7445 leader_election.cc:343] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a530070f493646f99d788c6da6ea3e23 (127.2.36.66:41537): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:52:57.023764 7546 raft_consensus.cc:513] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:57.029831 7539 tablet_bootstrap.cc:654] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:57.030995 7324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8b6c42fc174d5382f4cf9d0228dd33" candidate_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "252fc48398ea43f683ffb1ccfd923837"
I20250115 02:52:57.031666 7324 raft_consensus.cc:3054] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:57.033371 7546 leader_election.cc:290] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [CANDIDATE]: Term 1 election: Requested vote from peers 252fc48398ea43f683ffb1ccfd923837 (127.2.36.65:46765), a530070f493646f99d788c6da6ea3e23 (127.2.36.66:41537)
I20250115 02:52:57.038355 7324 raft_consensus.cc:2463] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcbc16aa7c0d44cf9eb8a24564749ece in term 1.
I20250115 02:52:57.040087 7445 leader_election.cc:304] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252fc48398ea43f683ffb1ccfd923837, bcbc16aa7c0d44cf9eb8a24564749ece; no voters:
I20250115 02:52:57.041515 7539 tablet_bootstrap.cc:492] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23: No bootstrap required, opened a new log
I20250115 02:52:57.042039 7539 ts_tablet_manager.cc:1397] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.002s
I20250115 02:52:57.043859 7546 raft_consensus.cc:2798] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:57.044785 7539 raft_consensus.cc:357] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } } peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } }
I20250115 02:52:57.045629 7539 raft_consensus.cc:383] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:57.045989 7539 raft_consensus.cc:738] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a530070f493646f99d788c6da6ea3e23, State: Initialized, Role: FOLLOWER
I20250115 02:52:57.046353 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8b6c42fc174d5382f4cf9d0228dd33" candidate_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a530070f493646f99d788c6da6ea3e23"
I20250115 02:52:57.045182 7546 raft_consensus.cc:695] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 LEADER]: Becoming Leader. State: Replica: bcbc16aa7c0d44cf9eb8a24564749ece, State: Running, Role: LEADER
I20250115 02:52:57.052435 7544 raft_consensus.cc:491] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:57.052894 7544 raft_consensus.cc:513] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } } peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } }
I20250115 02:52:57.047410 7546 consensus_queue.cc:237] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
W20250115 02:52:57.052037 7445 leader_election.cc:343] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a530070f493646f99d788c6da6ea3e23 (127.2.36.66:41537): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:52:57.055274 7544 leader_election.cc:290] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcbc16aa7c0d44cf9eb8a24564749ece (127.2.36.67:43115), a530070f493646f99d788c6da6ea3e23 (127.2.36.66:41537)
I20250115 02:52:57.046725 7539 consensus_queue.cc:260] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } } peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } }
I20250115 02:52:57.058621 7539 ts_tablet_manager.cc:1428] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23: Time spent starting tablet: real 0.016s user 0.001s sys 0.005s
I20250115 02:52:57.060585 7539 tablet_bootstrap.cc:492] T 3f8b6c42fc174d5382f4cf9d0228dd33 P a530070f493646f99d788c6da6ea3e23: Bootstrap starting.
I20250115 02:52:57.068284 7539 tablet_bootstrap.cc:654] T 3f8b6c42fc174d5382f4cf9d0228dd33 P a530070f493646f99d788c6da6ea3e23: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:57.076884 7199 catalog_manager.cc:5526] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece reported cstate change: term changed from 0 to 1, leader changed from <none> to bcbc16aa7c0d44cf9eb8a24564749ece (127.2.36.67). New cstate: current_term: 1 leader_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } health_report { overall_health: HEALTHY } } }
I20250115 02:52:57.106482 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcebc6c1cec4f1792bdd241fbc68a36" candidate_uuid: "252fc48398ea43f683ffb1ccfd923837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a530070f493646f99d788c6da6ea3e23" is_pre_election: true
I20250115 02:52:57.107003 7493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcebc6c1cec4f1792bdd241fbc68a36" candidate_uuid: "252fc48398ea43f683ffb1ccfd923837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" is_pre_election: true
I20250115 02:52:57.107275 7401 raft_consensus.cc:2463] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 252fc48398ea43f683ffb1ccfd923837 in term 0.
I20250115 02:52:57.107700 7493 raft_consensus.cc:2463] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 252fc48398ea43f683ffb1ccfd923837 in term 0.
I20250115 02:52:57.109489 7288 leader_election.cc:304] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252fc48398ea43f683ffb1ccfd923837, bcbc16aa7c0d44cf9eb8a24564749ece; no voters:
I20250115 02:52:57.111950 7544 raft_consensus.cc:2798] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:57.112325 7544 raft_consensus.cc:491] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:57.112634 7544 raft_consensus.cc:3054] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:57.121119 7544 raft_consensus.cc:513] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } } peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } }
I20250115 02:52:57.124650 7544 leader_election.cc:290] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [CANDIDATE]: Term 1 election: Requested vote from peers bcbc16aa7c0d44cf9eb8a24564749ece (127.2.36.67:43115), a530070f493646f99d788c6da6ea3e23 (127.2.36.66:41537)
I20250115 02:52:57.123208 7539 tablet_bootstrap.cc:492] T 3f8b6c42fc174d5382f4cf9d0228dd33 P a530070f493646f99d788c6da6ea3e23: No bootstrap required, opened a new log
I20250115 02:52:57.125550 7539 ts_tablet_manager.cc:1397] T 3f8b6c42fc174d5382f4cf9d0228dd33 P a530070f493646f99d788c6da6ea3e23: Time spent bootstrapping tablet: real 0.065s user 0.024s sys 0.020s
I20250115 02:52:57.128412 7539 raft_consensus.cc:357] T 3f8b6c42fc174d5382f4cf9d0228dd33 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:57.129168 7539 raft_consensus.cc:383] T 3f8b6c42fc174d5382f4cf9d0228dd33 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:57.129514 7539 raft_consensus.cc:738] T 3f8b6c42fc174d5382f4cf9d0228dd33 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a530070f493646f99d788c6da6ea3e23, State: Initialized, Role: FOLLOWER
I20250115 02:52:57.130394 7539 consensus_queue.cc:260] T 3f8b6c42fc174d5382f4cf9d0228dd33 P a530070f493646f99d788c6da6ea3e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } } peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } }
I20250115 02:52:57.137820 7493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcebc6c1cec4f1792bdd241fbc68a36" candidate_uuid: "252fc48398ea43f683ffb1ccfd923837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece"
I20250115 02:52:57.138253 7539 ts_tablet_manager.cc:1428] T 3f8b6c42fc174d5382f4cf9d0228dd33 P a530070f493646f99d788c6da6ea3e23: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250115 02:52:57.139062 7493 raft_consensus.cc:3054] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:57.141407 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcebc6c1cec4f1792bdd241fbc68a36" candidate_uuid: "252fc48398ea43f683ffb1ccfd923837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a530070f493646f99d788c6da6ea3e23"
I20250115 02:52:57.142017 7401 raft_consensus.cc:3054] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:57.148627 7401 raft_consensus.cc:2463] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 252fc48398ea43f683ffb1ccfd923837 in term 1.
I20250115 02:52:57.150673 7288 leader_election.cc:304] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252fc48398ea43f683ffb1ccfd923837, a530070f493646f99d788c6da6ea3e23; no voters:
I20250115 02:52:57.151382 7544 raft_consensus.cc:2798] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:57.153139 7544 raft_consensus.cc:695] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 1 LEADER]: Becoming Leader. State: Replica: 252fc48398ea43f683ffb1ccfd923837, State: Running, Role: LEADER
I20250115 02:52:57.154013 7544 consensus_queue.cc:237] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } } peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } }
I20250115 02:52:57.155802 7493 raft_consensus.cc:2463] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 252fc48398ea43f683ffb1ccfd923837 in term 1.
I20250115 02:52:57.172968 7199 catalog_manager.cc:5526] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 252fc48398ea43f683ffb1ccfd923837 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "252fc48398ea43f683ffb1ccfd923837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbc16aa7c0d44cf9eb8a24564749ece" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 43115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "252fc48398ea43f683ffb1ccfd923837" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 46765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a530070f493646f99d788c6da6ea3e23" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 41537 } health_report { overall_health: UNKNOWN } } }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/ksck_remote-test.cc:514: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20250115 02:52:57.325500 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:52:57.369905 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:57.370617 2193 tablet_replica.cc:331] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837: stopping tablet replica
I20250115 02:52:57.371215 2193 raft_consensus.cc:2238] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:57.371862 2193 pending_rounds.cc:62] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837: Trying to abort 1 pending ops.
I20250115 02:52:57.372116 2193 pending_rounds.cc:69] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114381628030799872 op_type: NO_OP noop_request { }
I20250115 02:52:57.372478 2193 raft_consensus.cc:2883] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 02:52:57.372812 2193 raft_consensus.cc:2267] T edcebc6c1cec4f1792bdd241fbc68a36 P 252fc48398ea43f683ffb1ccfd923837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:57.376725 2193 tablet_replica.cc:331] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837: stopping tablet replica
I20250115 02:52:57.377286 2193 raft_consensus.cc:2238] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:57.377697 2193 raft_consensus.cc:2267] T 3f8b6c42fc174d5382f4cf9d0228dd33 P 252fc48398ea43f683ffb1ccfd923837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:57.379772 2193 tablet_replica.cc:331] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837: stopping tablet replica
I20250115 02:52:57.380244 2193 raft_consensus.cc:2238] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:57.380589 2193 raft_consensus.cc:2267] T b487d84cdd4a49d78c81a671b8027339 P 252fc48398ea43f683ffb1ccfd923837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:57.397958 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:52:57.410910 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
W20250115 02:52:57.439893 7445 consensus_peers.cc:487] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece -> Peer a530070f493646f99d788c6da6ea3e23 (127.2.36.66:41537): Couldn't send request to peer a530070f493646f99d788c6da6ea3e23. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:57.445812 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:57.446578 2193 tablet_replica.cc:331] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23: stopping tablet replica
I20250115 02:52:57.447185 2193 raft_consensus.cc:2238] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:57.447599 2193 raft_consensus.cc:2267] T edcebc6c1cec4f1792bdd241fbc68a36 P a530070f493646f99d788c6da6ea3e23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:57.449492 2193 tablet_replica.cc:331] T 3f8b6c42fc174d5382f4cf9d0228dd33 P a530070f493646f99d788c6da6ea3e23: stopping tablet replica
I20250115 02:52:57.449932 2193 raft_consensus.cc:2238] T 3f8b6c42fc174d5382f4cf9d0228dd33 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:57.450229 2193 raft_consensus.cc:2267] T 3f8b6c42fc174d5382f4cf9d0228dd33 P a530070f493646f99d788c6da6ea3e23 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:57.451912 2193 tablet_replica.cc:331] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23: stopping tablet replica
I20250115 02:52:57.452704 2193 raft_consensus.cc:2238] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:57.453032 2193 raft_consensus.cc:2267] T b487d84cdd4a49d78c81a671b8027339 P a530070f493646f99d788c6da6ea3e23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:57.475056 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
W20250115 02:52:57.483511 7445 consensus_peers.cc:487] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece -> Peer 252fc48398ea43f683ffb1ccfd923837 (127.2.36.65:46765): Couldn't send request to peer 252fc48398ea43f683ffb1ccfd923837. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:46765: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:52:57.486861 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:52:57.508074 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:52:57.508657 2193 tablet_replica.cc:331] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece: stopping tablet replica
I20250115 02:52:57.509238 2193 raft_consensus.cc:2238] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:57.509604 2193 raft_consensus.cc:2267] T edcebc6c1cec4f1792bdd241fbc68a36 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:57.511294 2193 tablet_replica.cc:331] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece: stopping tablet replica
I20250115 02:52:57.511775 2193 raft_consensus.cc:2238] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:57.512352 2193 pending_rounds.cc:62] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece: Trying to abort 1 pending ops.
I20250115 02:52:57.512529 2193 pending_rounds.cc:69] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114381627618615296 op_type: NO_OP noop_request { }
I20250115 02:52:57.512761 2193 raft_consensus.cc:2883] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 02:52:57.512981 2193 raft_consensus.cc:2267] T 3f8b6c42fc174d5382f4cf9d0228dd33 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:57.514959 2193 tablet_replica.cc:331] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece: stopping tablet replica
I20250115 02:52:57.515395 2193 raft_consensus.cc:2238] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:57.515959 2193 pending_rounds.cc:62] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece: Trying to abort 1 pending ops.
I20250115 02:52:57.516117 2193 pending_rounds.cc:69] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114381627309572096 op_type: NO_OP noop_request { }
I20250115 02:52:57.516343 2193 raft_consensus.cc:2883] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 02:52:57.516566 2193 raft_consensus.cc:2267] T b487d84cdd4a49d78c81a671b8027339 P bcbc16aa7c0d44cf9eb8a24564749ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:57.537956 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:52:57.548400 2193 master.cc:537] Master@127.2.36.126:33081 shutting down...
I20250115 02:52:57.564675 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:57.565192 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:57.565505 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 4c56a121def049488a306eace8ffebb6: stopping tablet replica
I20250115 02:52:57.584964 2193 master.cc:559] Master@127.2.36.126:33081 shutdown complete.
I20250115 02:52:57.596324 2193 master.cc:537] Master@127.2.36.125:38391 shutting down...
I20250115 02:52:57.609498 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:52:57.609998 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:57.610289 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 11c08d58238c49449fcba36cf3f55fbb: stopping tablet replica
I20250115 02:52:57.628772 2193 master.cc:559] Master@127.2.36.125:38391 shutdown complete.
I20250115 02:52:57.639926 2193 master.cc:537] Master@127.2.36.124:41773 shutting down...
I20250115 02:52:57.652709 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:52:57.653477 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:52:57.653789 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2595e4cdffa24599aa59225bb41bdb25: stopping tablet replica
I20250115 02:52:57.671746 2193 master.cc:559] Master@127.2.36.124:41773 shutdown complete.
[ SKIPPED ] RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM (2139 ms)
[ RUN ] RemoteKsckTest.TestLeaderMasterDown
I20250115 02:52:57.696946 2193 test_util.cc:274] Using random seed: -966609252
I20250115 02:52:57.697870 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:38529,127.2.36.125:41901,127.2.36.124:45399
I20250115 02:52:57.699131 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:57.704444 7567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:57.705278 7568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:57.706110 7570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:57.706724 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:57.707432 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:57.707604 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:57.707711 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909577707701 us; error 0 us; skew 500 ppm
I20250115 02:52:57.708150 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:57.710216 2193 webserver.cc:458] Webserver started at http://127.2.36.126:40631/ using document root <none> and password file <none>
I20250115 02:52:57.710613 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:57.710808 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:57.711028 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:57.712000 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "87f997285c2c42d48e14a596166dcb4b"
format_stamp: "Formatted at 2025-01-15 02:52:57 on dist-test-slave-1lv2"
I20250115 02:52:57.716048 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:52:57.719115 7575 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:57.719905 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:52:57.720191 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "87f997285c2c42d48e14a596166dcb4b"
format_stamp: "Formatted at 2025-01-15 02:52:57 on dist-test-slave-1lv2"
I20250115 02:52:57.720446 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:57.749684 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:57.750664 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:57.782868 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:38529
I20250115 02:52:57.782967 7626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:38529 every 8 connection(s)
I20250115 02:52:57.786345 7627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:57.786713 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:57.792693 7629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:57.793219 7630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:57.793292 7627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:57.795599 7632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:57.797276 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:57.798012 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:57.798209 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:57.798349 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909577798334 us; error 0 us; skew 500 ppm
I20250115 02:52:57.798908 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:57.801384 2193 webserver.cc:458] Webserver started at http://127.2.36.125:41691/ using document root <none> and password file <none>
I20250115 02:52:57.801807 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:57.801977 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:57.802217 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:57.803349 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "86b72de984ad4484a7f26a5d0377e575"
format_stamp: "Formatted at 2025-01-15 02:52:57 on dist-test-slave-1lv2"
I20250115 02:52:57.805953 7627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:57.808102 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
W20250115 02:52:57.810714 7627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:41901: Network error: Client connection negotiation failed: client connection to 127.2.36.125:41901: connect: Connection refused (error 111)
I20250115 02:52:57.811512 7640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:57.812215 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20250115 02:52:57.812470 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "86b72de984ad4484a7f26a5d0377e575"
format_stamp: "Formatted at 2025-01-15 02:52:57 on dist-test-slave-1lv2"
I20250115 02:52:57.812717 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:57.840548 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:57.841521 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:57.843308 7627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } attempt: 1
W20250115 02:52:57.847239 7627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:41901: Network error: Client connection negotiation failed: client connection to 127.2.36.125:41901: connect: Connection refused (error 111)
I20250115 02:52:57.875161 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:41901
I20250115 02:52:57.875294 7691 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:41901 every 8 connection(s)
I20250115 02:52:57.878894 7692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:57.879021 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:52:57.884817 7694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:57.885783 7695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:57.886510 7697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:57.886529 7692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:57.888976 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:57.889976 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:57.890249 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:57.890447 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909577890427 us; error 0 us; skew 500 ppm
I20250115 02:52:57.891048 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:57.893611 2193 webserver.cc:458] Webserver started at http://127.2.36.124:38705/ using document root <none> and password file <none>
I20250115 02:52:57.894097 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:57.894341 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:57.894639 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:57.895992 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "a77a4fffef954686be239477d1274168"
format_stamp: "Formatted at 2025-01-15 02:52:57 on dist-test-slave-1lv2"
I20250115 02:52:57.897436 7692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:57.901281 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250115 02:52:57.902630 7627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } attempt: 2
I20250115 02:52:57.906148 7705 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:57.907152 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250115 02:52:57.907567 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "a77a4fffef954686be239477d1274168"
format_stamp: "Formatted at 2025-01-15 02:52:57 on dist-test-slave-1lv2"
I20250115 02:52:57.907933 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:57.909840 7692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:57.916081 7692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45399: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45399: connect: Connection refused (error 111)
I20250115 02:52:57.917829 7627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:52:57.921762 7627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45399: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45399: connect: Connection refused (error 111)
I20250115 02:52:57.936765 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:57.937808 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:57.943181 7627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } attempt: 1
W20250115 02:52:57.947301 7627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45399: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45399: connect: Connection refused (error 111)
I20250115 02:52:57.951531 7692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } attempt: 1
W20250115 02:52:57.955832 7692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45399: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45399: connect: Connection refused (error 111)
I20250115 02:52:57.974452 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:45399
I20250115 02:52:57.974565 7757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:45399 every 8 connection(s)
I20250115 02:52:57.977417 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:52:57.978318 7758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:57.984320 7758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:57.994201 7758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:58.001279 7692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } attempt: 2
I20250115 02:52:58.003129 7758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:52:58.015807 7627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } attempt: 2
I20250115 02:52:58.017910 7692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575: Bootstrap starting.
I20250115 02:52:58.019876 7758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168: Bootstrap starting.
I20250115 02:52:58.023250 7692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:58.025718 7758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:58.028066 7692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575: No bootstrap required, opened a new log
I20250115 02:52:58.030018 7758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168: No bootstrap required, opened a new log
I20250115 02:52:58.030799 7692 raft_consensus.cc:357] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
I20250115 02:52:58.031558 7692 raft_consensus.cc:383] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:58.031837 7692 raft_consensus.cc:738] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b72de984ad4484a7f26a5d0377e575, State: Initialized, Role: FOLLOWER
I20250115 02:52:58.032063 7758 raft_consensus.cc:357] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
I20250115 02:52:58.032708 7758 raft_consensus.cc:383] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:58.032964 7758 raft_consensus.cc:738] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77a4fffef954686be239477d1274168, State: Initialized, Role: FOLLOWER
I20250115 02:52:58.032531 7692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
I20250115 02:52:58.033532 7758 consensus_queue.cc:260] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
I20250115 02:52:58.035112 7627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b: Bootstrap starting.
I20250115 02:52:58.035297 7766 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:52:58.036127 7766 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:58.037071 7758 sys_catalog.cc:564] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:58.037194 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:52:58.037972 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:58.038895 7692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:52:58.041330 7627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:58.050029 7627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b: No bootstrap required, opened a new log
I20250115 02:52:58.052896 7627 raft_consensus.cc:357] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
I20250115 02:52:58.053786 7627 raft_consensus.cc:383] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:58.054159 7627 raft_consensus.cc:738] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f997285c2c42d48e14a596166dcb4b, State: Initialized, Role: FOLLOWER
I20250115 02:52:58.054936 7627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
I20250115 02:52:58.057173 7786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:52:58.058019 7786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:58.059201 7627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [sys.catalog]: configured and running, proceeding with master startup.
W20250115 02:52:58.064007 7787 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:58.064415 7787 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:52:58.067847 7792 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250115 02:52:58.067802 7766 raft_consensus.cc:491] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 02:52:58.068331 7792 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:52:58.068545 7766 raft_consensus.cc:513] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
I20250115 02:52:58.070549 7766 leader_election.cc:290] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87f997285c2c42d48e14a596166dcb4b (127.2.36.126:38529), 86b72de984ad4484a7f26a5d0377e575 (127.2.36.125:41901)
I20250115 02:52:58.071585 7602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a77a4fffef954686be239477d1274168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f997285c2c42d48e14a596166dcb4b" is_pre_election: true
I20250115 02:52:58.071848 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a77a4fffef954686be239477d1274168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86b72de984ad4484a7f26a5d0377e575" is_pre_election: true
I20250115 02:52:58.072420 7602 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a77a4fffef954686be239477d1274168 in term 0.
I20250115 02:52:58.072528 7667 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a77a4fffef954686be239477d1274168 in term 0.
I20250115 02:52:58.073239 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:52:58.073504 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:52:58.073802 7710 leader_election.cc:304] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b72de984ad4484a7f26a5d0377e575, a77a4fffef954686be239477d1274168; no voters:
I20250115 02:52:58.075002 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:58.074926 7766 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:58.075470 7766 raft_consensus.cc:491] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:58.075804 7766 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 0 FOLLOWER]: Advancing to term 1
W20250115 02:52:58.077234 7801 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:52:58.077548 7801 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:52:58.082216 7802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:58.082460 7766 raft_consensus.cc:513] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
W20250115 02:52:58.083293 7803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:58.085413 7766 leader_election.cc:290] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [CANDIDATE]: Term 1 election: Requested vote from peers 87f997285c2c42d48e14a596166dcb4b (127.2.36.126:38529), 86b72de984ad4484a7f26a5d0377e575 (127.2.36.125:41901)
I20250115 02:52:58.085587 7602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a77a4fffef954686be239477d1274168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f997285c2c42d48e14a596166dcb4b"
I20250115 02:52:58.086109 7602 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 0 FOLLOWER]: Advancing to term 1
W20250115 02:52:58.086490 7805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:58.087287 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:58.087914 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a77a4fffef954686be239477d1274168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86b72de984ad4484a7f26a5d0377e575"
I20250115 02:52:58.088284 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:58.088603 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:58.088878 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909578088860 us; error 0 us; skew 500 ppm
I20250115 02:52:58.089484 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:58.088511 7667 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:58.091584 7602 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a77a4fffef954686be239477d1274168 in term 1.
I20250115 02:52:58.092432 2193 webserver.cc:458] Webserver started at http://127.2.36.65:40245/ using document root <none> and password file <none>
I20250115 02:52:58.092550 7709 leader_election.cc:304] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87f997285c2c42d48e14a596166dcb4b, a77a4fffef954686be239477d1274168; no voters:
I20250115 02:52:58.093114 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:58.093277 7766 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:58.093439 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:58.093879 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:58.094875 7766 raft_consensus.cc:695] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 1 LEADER]: Becoming Leader. State: Replica: a77a4fffef954686be239477d1274168, State: Running, Role: LEADER
I20250115 02:52:58.095276 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "7013afe870964e1693fe739e7c7a4ad2"
format_stamp: "Formatted at 2025-01-15 02:52:58 on dist-test-slave-1lv2"
I20250115 02:52:58.095566 7766 consensus_queue.cc:237] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
I20250115 02:52:58.095991 7667 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a77a4fffef954686be239477d1274168 in term 1.
I20250115 02:52:58.100279 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a77a4fffef954686be239477d1274168. Latest consensus state: current_term: 1 leader_uuid: "a77a4fffef954686be239477d1274168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:52:58.100832 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250115 02:52:58.100958 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:58.102341 7812 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:52:58.104701 7813 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:58.105501 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250115 02:52:58.105767 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "7013afe870964e1693fe739e7c7a4ad2"
format_stamp: "Formatted at 2025-01-15 02:52:58 on dist-test-slave-1lv2"
I20250115 02:52:58.106029 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:58.107693 7812 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:52:58.115979 7667 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 1 FOLLOWER]: Refusing update from remote peer a77a4fffef954686be239477d1274168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:58.116130 7602 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 1 FOLLOWER]: Refusing update from remote peer a77a4fffef954686be239477d1274168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:52:58.117230 7766 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:58.117878 7809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:58.127420 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:58.128800 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:58.129632 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a77a4fffef954686be239477d1274168. Latest consensus state: current_term: 1 leader_uuid: "a77a4fffef954686be239477d1274168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:52:58.130486 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:58.130241 7786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a77a4fffef954686be239477d1274168. Latest consensus state: current_term: 1 leader_uuid: "a77a4fffef954686be239477d1274168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:52:58.130970 7786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:58.131534 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:58.134668 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:58.134985 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250115 02:52:58.135354 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:58.135627 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:58.146543 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a77a4fffef954686be239477d1274168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:52:58.147450 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:58.147738 7766 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a77a4fffef954686be239477d1274168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:52:58.148085 7786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a77a4fffef954686be239477d1274168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:52:58.148527 7766 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:58.148753 7786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:52:58.149698 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a77a4fffef954686be239477d1274168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:52:58.150506 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [sys.catalog]: This master's current role is: LEADER
I20250115 02:52:58.152982 7812 catalog_manager.cc:1348] Generated new cluster ID: 388c38c4263e44329a79a6b325b50d71
I20250115 02:52:58.153301 7812 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:52:58.194416 7812 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:52:58.196038 7812 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:52:58.208914 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:33371
I20250115 02:52:58.209015 7884 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:33371 every 8 connection(s)
I20250115 02:52:58.223358 7812 catalog_manager.cc:5899] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168: Generated new TSK 0
I20250115 02:52:58.224174 7812 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:52:58.244199 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:58.246814 7886 heartbeater.cc:346] Connected to a master server at 127.2.36.125:41901
I20250115 02:52:58.247259 7886 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:58.247395 7885 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45399
I20250115 02:52:58.247723 7885 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:58.248109 7886 heartbeater.cc:509] Master 127.2.36.125:41901 requested a full tablet report, sending...
I20250115 02:52:58.248445 7885 heartbeater.cc:509] Master 127.2.36.124:45399 requested a full tablet report, sending...
I20250115 02:52:58.250926 7657 ts_manager.cc:194] Registered new tserver with Master: 7013afe870964e1693fe739e7c7a4ad2 (127.2.36.65:33371)
I20250115 02:52:58.251156 7723 ts_manager.cc:194] Registered new tserver with Master: 7013afe870964e1693fe739e7c7a4ad2 (127.2.36.65:33371)
I20250115 02:52:58.253479 7723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49934
I20250115 02:52:58.254840 7888 heartbeater.cc:346] Connected to a master server at 127.2.36.126:38529
I20250115 02:52:58.255290 7888 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:58.255999 7888 heartbeater.cc:509] Master 127.2.36.126:38529 requested a full tablet report, sending...
W20250115 02:52:58.258672 7893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:52:58.260392 7895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:58.260877 7592 ts_manager.cc:194] Registered new tserver with Master: 7013afe870964e1693fe739e7c7a4ad2 (127.2.36.65:33371)
W20250115 02:52:58.260937 7892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:58.262301 2193 server_base.cc:1034] running on GCE node
I20250115 02:52:58.263095 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:58.263306 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:58.263484 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909578263467 us; error 0 us; skew 500 ppm
I20250115 02:52:58.263964 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:58.266258 2193 webserver.cc:458] Webserver started at http://127.2.36.66:33389/ using document root <none> and password file <none>
I20250115 02:52:58.266438 7885 heartbeater.cc:501] Master 127.2.36.124:45399 was elected leader, sending a full tablet report...
I20250115 02:52:58.266708 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:58.266983 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:58.267241 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:58.268518 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "1a58e5d41bbf48f1848bbbea7998b645"
format_stamp: "Formatted at 2025-01-15 02:52:58 on dist-test-slave-1lv2"
I20250115 02:52:58.272858 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:52:58.276728 7900 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:58.277549 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:52:58.277869 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "1a58e5d41bbf48f1848bbbea7998b645"
format_stamp: "Formatted at 2025-01-15 02:52:58 on dist-test-slave-1lv2"
I20250115 02:52:58.278108 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:58.293035 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:58.294157 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:58.295778 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:58.297904 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:58.298115 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:58.298326 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:58.298478 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:58.342100 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:46219
I20250115 02:52:58.342315 7962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:46219 every 8 connection(s)
I20250115 02:52:58.358084 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:52:58.371506 7964 heartbeater.cc:346] Connected to a master server at 127.2.36.125:41901
I20250115 02:52:58.371982 7964 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:58.372064 7963 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45399
I20250115 02:52:58.372509 7963 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:58.372970 7964 heartbeater.cc:509] Master 127.2.36.125:41901 requested a full tablet report, sending...
I20250115 02:52:58.373302 7963 heartbeater.cc:509] Master 127.2.36.124:45399 requested a full tablet report, sending...
I20250115 02:52:58.375412 7723 ts_manager.cc:194] Registered new tserver with Master: 1a58e5d41bbf48f1848bbbea7998b645 (127.2.36.66:46219)
W20250115 02:52:58.375736 7970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:58.375941 7657 ts_manager.cc:194] Registered new tserver with Master: 1a58e5d41bbf48f1848bbbea7998b645 (127.2.36.66:46219)
I20250115 02:52:58.377195 7723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49938
W20250115 02:52:58.379527 7971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:58.385941 2193 server_base.cc:1034] running on GCE node
W20250115 02:52:58.386735 7973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:52:58.387199 7965 heartbeater.cc:346] Connected to a master server at 127.2.36.126:38529
I20250115 02:52:58.387627 7965 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:58.387670 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:52:58.388057 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:52:58.388231 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909578388213 us; error 0 us; skew 500 ppm
I20250115 02:52:58.388375 7965 heartbeater.cc:509] Master 127.2.36.126:38529 requested a full tablet report, sending...
I20250115 02:52:58.388813 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:52:58.389709 7963 heartbeater.cc:501] Master 127.2.36.124:45399 was elected leader, sending a full tablet report...
I20250115 02:52:58.392223 7592 ts_manager.cc:194] Registered new tserver with Master: 1a58e5d41bbf48f1848bbbea7998b645 (127.2.36.66:46219)
I20250115 02:52:58.395206 2193 webserver.cc:458] Webserver started at http://127.2.36.67:36713/ using document root <none> and password file <none>
I20250115 02:52:58.395875 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:52:58.396106 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:52:58.396355 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:52:58.397334 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "36b1e5772ad44039b76198414b571cf1"
format_stamp: "Formatted at 2025-01-15 02:52:58 on dist-test-slave-1lv2"
I20250115 02:52:58.401711 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20250115 02:52:58.407220 7978 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:58.409374 2193 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.001s
I20250115 02:52:58.409780 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "36b1e5772ad44039b76198414b571cf1"
format_stamp: "Formatted at 2025-01-15 02:52:58 on dist-test-slave-1lv2"
I20250115 02:52:58.410032 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestLeaderMasterDown.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:52:58.421399 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:52:58.423005 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:52:58.424252 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:52:58.426563 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:52:58.426771 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:58.426990 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:52:58.427140 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:52:58.484879 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:33757
I20250115 02:52:58.484984 8040 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:33757 every 8 connection(s)
I20250115 02:52:58.517104 8042 heartbeater.cc:346] Connected to a master server at 127.2.36.125:41901
I20250115 02:52:58.517607 8042 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:58.518520 8042 heartbeater.cc:509] Master 127.2.36.125:41901 requested a full tablet report, sending...
I20250115 02:52:58.530807 7657 ts_manager.cc:194] Registered new tserver with Master: 36b1e5772ad44039b76198414b571cf1 (127.2.36.67:33757)
I20250115 02:52:58.537715 8044 heartbeater.cc:346] Connected to a master server at 127.2.36.126:38529
I20250115 02:52:58.538113 8044 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:58.538568 8041 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45399
I20250115 02:52:58.539000 8041 heartbeater.cc:463] Registering TS with master...
I20250115 02:52:58.538992 8044 heartbeater.cc:509] Master 127.2.36.126:38529 requested a full tablet report, sending...
I20250115 02:52:58.540006 8041 heartbeater.cc:509] Master 127.2.36.124:45399 requested a full tablet report, sending...
I20250115 02:52:58.542085 7722 ts_manager.cc:194] Registered new tserver with Master: 36b1e5772ad44039b76198414b571cf1 (127.2.36.67:33757)
I20250115 02:52:58.542327 7591 ts_manager.cc:194] Registered new tserver with Master: 36b1e5772ad44039b76198414b571cf1 (127.2.36.67:33757)
I20250115 02:52:58.543421 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007674088s
I20250115 02:52:58.543871 7722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49954
I20250115 02:52:58.556942 8041 heartbeater.cc:501] Master 127.2.36.124:45399 was elected leader, sending a full tablet report...
I20250115 02:52:58.598881 7722 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49966:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:52:58.601289 7722 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:52:58.710443 7927 tablet_service.cc:1467] Processing CreateTablet for tablet e4118be3dcfc49bf8a7afe9210e91e9e (DEFAULT_TABLE table=ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:58.710655 7926 tablet_service.cc:1467] Processing CreateTablet for tablet e533b7b65683437aa22299a11437290f (DEFAULT_TABLE table=ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:58.710655 7928 tablet_service.cc:1467] Processing CreateTablet for tablet efe29696b7a4432a918ed169149c1c41 (DEFAULT_TABLE table=ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:58.712980 7927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4118be3dcfc49bf8a7afe9210e91e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:58.722270 7928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efe29696b7a4432a918ed169149c1c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:58.727713 7926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e533b7b65683437aa22299a11437290f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:58.769176 8062 tablet_bootstrap.cc:492] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645: Bootstrap starting.
I20250115 02:52:58.782047 8062 tablet_bootstrap.cc:654] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:58.783001 7850 tablet_service.cc:1467] Processing CreateTablet for tablet efe29696b7a4432a918ed169149c1c41 (DEFAULT_TABLE table=ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:58.784322 7850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efe29696b7a4432a918ed169149c1c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:58.796350 7849 tablet_service.cc:1467] Processing CreateTablet for tablet e4118be3dcfc49bf8a7afe9210e91e9e (DEFAULT_TABLE table=ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:58.797829 7849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4118be3dcfc49bf8a7afe9210e91e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:58.811811 8006 tablet_service.cc:1467] Processing CreateTablet for tablet efe29696b7a4432a918ed169149c1c41 (DEFAULT_TABLE table=ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:52:58.813408 8006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efe29696b7a4432a918ed169149c1c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:58.822427 8005 tablet_service.cc:1467] Processing CreateTablet for tablet e4118be3dcfc49bf8a7afe9210e91e9e (DEFAULT_TABLE table=ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:52:58.824142 8005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4118be3dcfc49bf8a7afe9210e91e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:58.850646 8004 tablet_service.cc:1467] Processing CreateTablet for tablet e533b7b65683437aa22299a11437290f (DEFAULT_TABLE table=ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:58.851869 8004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e533b7b65683437aa22299a11437290f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:58.875231 7848 tablet_service.cc:1467] Processing CreateTablet for tablet e533b7b65683437aa22299a11437290f (DEFAULT_TABLE table=ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:52:58.876744 7848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e533b7b65683437aa22299a11437290f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:52:58.903430 8064 tablet_bootstrap.cc:492] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1: Bootstrap starting.
I20250115 02:52:58.909333 8062 tablet_bootstrap.cc:492] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645: No bootstrap required, opened a new log
I20250115 02:52:58.909708 8065 tablet_bootstrap.cc:492] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2: Bootstrap starting.
I20250115 02:52:58.909941 8062 ts_tablet_manager.cc:1397] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645: Time spent bootstrapping tablet: real 0.141s user 0.050s sys 0.078s
I20250115 02:52:58.914619 8062 raft_consensus.cc:357] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } } peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } }
I20250115 02:52:58.915410 8062 raft_consensus.cc:383] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:58.915715 8062 raft_consensus.cc:738] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a58e5d41bbf48f1848bbbea7998b645, State: Initialized, Role: FOLLOWER
I20250115 02:52:58.915922 8065 tablet_bootstrap.cc:654] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:58.920078 8064 tablet_bootstrap.cc:654] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:58.926848 8065 tablet_bootstrap.cc:492] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2: No bootstrap required, opened a new log
I20250115 02:52:58.927438 8065 ts_tablet_manager.cc:1397] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20250115 02:52:58.929805 8062 consensus_queue.cc:260] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } } peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } }
I20250115 02:52:58.930063 8065 raft_consensus.cc:357] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } } peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } }
I20250115 02:52:58.931175 8065 raft_consensus.cc:383] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:58.931540 8065 raft_consensus.cc:738] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7013afe870964e1693fe739e7c7a4ad2, State: Initialized, Role: FOLLOWER
I20250115 02:52:58.932343 8065 consensus_queue.cc:260] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } } peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } }
I20250115 02:52:58.936878 8062 ts_tablet_manager.cc:1428] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645: Time spent starting tablet: real 0.025s user 0.009s sys 0.000s
I20250115 02:52:58.939793 8062 tablet_bootstrap.cc:492] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645: Bootstrap starting.
I20250115 02:52:58.942025 8065 ts_tablet_manager.cc:1428] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2: Time spent starting tablet: real 0.014s user 0.005s sys 0.003s
I20250115 02:52:58.943176 8065 tablet_bootstrap.cc:492] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2: Bootstrap starting.
I20250115 02:52:58.945560 8062 tablet_bootstrap.cc:654] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:58.953126 8065 tablet_bootstrap.cc:654] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:58.956305 8064 tablet_bootstrap.cc:492] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1: No bootstrap required, opened a new log
I20250115 02:52:58.956923 8064 ts_tablet_manager.cc:1397] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.008s
I20250115 02:52:58.959825 8064 raft_consensus.cc:357] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:58.960906 8064 raft_consensus.cc:383] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:58.961299 8064 raft_consensus.cc:738] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b1e5772ad44039b76198414b571cf1, State: Initialized, Role: FOLLOWER
I20250115 02:52:58.968336 8065 tablet_bootstrap.cc:492] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2: No bootstrap required, opened a new log
I20250115 02:52:58.962162 8064 consensus_queue.cc:260] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:58.968972 8065 ts_tablet_manager.cc:1397] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.005s
I20250115 02:52:58.972544 8065 raft_consensus.cc:357] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:58.973405 8065 raft_consensus.cc:383] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:58.973745 8065 raft_consensus.cc:738] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7013afe870964e1693fe739e7c7a4ad2, State: Initialized, Role: FOLLOWER
I20250115 02:52:58.974553 8065 consensus_queue.cc:260] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:58.977010 8065 ts_tablet_manager.cc:1428] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250115 02:52:58.983573 8065 tablet_bootstrap.cc:492] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2: Bootstrap starting.
I20250115 02:52:58.985059 8062 tablet_bootstrap.cc:492] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645: No bootstrap required, opened a new log
I20250115 02:52:58.985611 8062 ts_tablet_manager.cc:1397] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645: Time spent bootstrapping tablet: real 0.046s user 0.018s sys 0.013s
I20250115 02:52:58.988330 8062 raft_consensus.cc:357] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:58.989089 8062 raft_consensus.cc:383] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:58.989382 8062 raft_consensus.cc:738] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a58e5d41bbf48f1848bbbea7998b645, State: Initialized, Role: FOLLOWER
I20250115 02:52:58.989794 8065 tablet_bootstrap.cc:654] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:58.990046 8062 consensus_queue.cc:260] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:58.998436 8062 ts_tablet_manager.cc:1428] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250115 02:52:58.999558 8062 tablet_bootstrap.cc:492] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645: Bootstrap starting.
I20250115 02:52:59.005335 8062 tablet_bootstrap.cc:654] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:59.033417 8062 tablet_bootstrap.cc:492] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645: No bootstrap required, opened a new log
I20250115 02:52:59.033931 8062 ts_tablet_manager.cc:1397] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.000s
I20250115 02:52:59.036749 8062 raft_consensus.cc:357] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.037142 8068 raft_consensus.cc:491] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:59.037497 8062 raft_consensus.cc:383] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:59.037824 8062 raft_consensus.cc:738] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a58e5d41bbf48f1848bbbea7998b645, State: Initialized, Role: FOLLOWER
I20250115 02:52:59.037626 8068 raft_consensus.cc:513] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } } peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } }
I20250115 02:52:59.038537 8062 consensus_queue.cc:260] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.040233 8068 leader_election.cc:290] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36b1e5772ad44039b76198414b571cf1 (127.2.36.67:33757), 7013afe870964e1693fe739e7c7a4ad2 (127.2.36.65:33371)
I20250115 02:52:59.075704 8064 ts_tablet_manager.cc:1428] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1: Time spent starting tablet: real 0.118s user 0.042s sys 0.038s
I20250115 02:52:59.076912 8064 tablet_bootstrap.cc:492] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1: Bootstrap starting.
I20250115 02:52:59.079818 7787 catalog_manager.cc:1260] Loaded cluster ID: 388c38c4263e44329a79a6b325b50d71
I20250115 02:52:59.080261 7787 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575: loading cluster ID for follower catalog manager: success
I20250115 02:52:59.083140 7801 catalog_manager.cc:1260] Loaded cluster ID: 388c38c4263e44329a79a6b325b50d71
I20250115 02:52:59.083503 7801 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b: loading cluster ID for follower catalog manager: success
I20250115 02:52:59.090626 8062 ts_tablet_manager.cc:1428] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645: Time spent starting tablet: real 0.056s user 0.014s sys 0.000s
I20250115 02:52:59.095736 8016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe29696b7a4432a918ed169149c1c41" candidate_uuid: "1a58e5d41bbf48f1848bbbea7998b645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b1e5772ad44039b76198414b571cf1" is_pre_election: true
I20250115 02:52:59.103641 7801 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b: acquiring CA information for follower catalog manager: success
W20250115 02:52:59.106180 7901 leader_election.cc:343] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 36b1e5772ad44039b76198414b571cf1 (127.2.36.67:33757): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:52:59.106688 8064 tablet_bootstrap.cc:654] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:59.112057 7801 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:59.117945 7860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe29696b7a4432a918ed169149c1c41" candidate_uuid: "1a58e5d41bbf48f1848bbbea7998b645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7013afe870964e1693fe739e7c7a4ad2" is_pre_election: true
I20250115 02:52:59.118899 7860 raft_consensus.cc:2463] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a58e5d41bbf48f1848bbbea7998b645 in term 0.
I20250115 02:52:59.120296 7901 leader_election.cc:304] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a58e5d41bbf48f1848bbbea7998b645, 7013afe870964e1693fe739e7c7a4ad2; no voters: 36b1e5772ad44039b76198414b571cf1
I20250115 02:52:59.121380 8075 raft_consensus.cc:2798] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:59.121850 8075 raft_consensus.cc:491] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:59.122151 8075 raft_consensus.cc:3054] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:59.128814 8064 tablet_bootstrap.cc:492] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1: No bootstrap required, opened a new log
I20250115 02:52:59.129374 8064 ts_tablet_manager.cc:1397] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.003s
I20250115 02:52:59.132404 8064 raft_consensus.cc:357] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } } peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } }
I20250115 02:52:59.133267 8064 raft_consensus.cc:383] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:59.133605 8064 raft_consensus.cc:738] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b1e5772ad44039b76198414b571cf1, State: Initialized, Role: FOLLOWER
I20250115 02:52:59.133898 8075 raft_consensus.cc:513] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } } peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } }
I20250115 02:52:59.134477 8064 consensus_queue.cc:260] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } } peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } }
I20250115 02:52:59.135910 7787 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575: acquiring CA information for follower catalog manager: success
I20250115 02:52:59.136808 8016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe29696b7a4432a918ed169149c1c41" candidate_uuid: "1a58e5d41bbf48f1848bbbea7998b645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b1e5772ad44039b76198414b571cf1"
I20250115 02:52:59.137408 8016 raft_consensus.cc:3054] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:59.140997 8064 ts_tablet_manager.cc:1428] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250115 02:52:59.141988 8064 tablet_bootstrap.cc:492] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1: Bootstrap starting.
I20250115 02:52:59.146008 8016 raft_consensus.cc:2463] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a58e5d41bbf48f1848bbbea7998b645 in term 1.
I20250115 02:52:59.147686 8064 tablet_bootstrap.cc:654] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1: Neither blocks nor log segments found. Creating new log.
I20250115 02:52:59.154150 8064 tablet_bootstrap.cc:492] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1: No bootstrap required, opened a new log
I20250115 02:52:59.154711 8064 ts_tablet_manager.cc:1397] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20250115 02:52:59.157454 8064 raft_consensus.cc:357] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.158226 8064 raft_consensus.cc:383] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:59.158573 8064 raft_consensus.cc:738] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b1e5772ad44039b76198414b571cf1, State: Initialized, Role: FOLLOWER
I20250115 02:52:59.159416 8064 consensus_queue.cc:260] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.162384 8064 ts_tablet_manager.cc:1428] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250115 02:52:59.168035 7901 leader_election.cc:304] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a58e5d41bbf48f1848bbbea7998b645, 36b1e5772ad44039b76198414b571cf1; no voters:
I20250115 02:52:59.168953 8068 raft_consensus.cc:2798] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:59.170982 8068 raft_consensus.cc:695] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 LEADER]: Becoming Leader. State: Replica: 1a58e5d41bbf48f1848bbbea7998b645, State: Running, Role: LEADER
I20250115 02:52:59.171671 8068 consensus_queue.cc:237] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } } peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } }
I20250115 02:52:59.166662 7787 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:52:59.180325 7860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe29696b7a4432a918ed169149c1c41" candidate_uuid: "1a58e5d41bbf48f1848bbbea7998b645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7013afe870964e1693fe739e7c7a4ad2"
I20250115 02:52:59.179684 7722 catalog_manager.cc:5526] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a58e5d41bbf48f1848bbbea7998b645 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "1a58e5d41bbf48f1848bbbea7998b645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:59.180915 7860 raft_consensus.cc:3054] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:59.185755 8075 leader_election.cc:290] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [CANDIDATE]: Term 1 election: Requested vote from peers 36b1e5772ad44039b76198414b571cf1 (127.2.36.67:33757), 7013afe870964e1693fe739e7c7a4ad2 (127.2.36.65:33371)
I20250115 02:52:59.191089 8065 tablet_bootstrap.cc:492] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2: No bootstrap required, opened a new log
I20250115 02:52:59.189296 7860 raft_consensus.cc:2463] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a58e5d41bbf48f1848bbbea7998b645 in term 1.
I20250115 02:52:59.191527 8065 ts_tablet_manager.cc:1397] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2: Time spent bootstrapping tablet: real 0.214s user 0.033s sys 0.022s
I20250115 02:52:59.194132 8065 raft_consensus.cc:357] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.194847 8065 raft_consensus.cc:383] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:52:59.195125 8065 raft_consensus.cc:738] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7013afe870964e1693fe739e7c7a4ad2, State: Initialized, Role: FOLLOWER
I20250115 02:52:59.195930 8065 consensus_queue.cc:260] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.206830 8065 ts_tablet_manager.cc:1428] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250115 02:52:59.299283 8072 raft_consensus.cc:491] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:59.302071 8072 raft_consensus.cc:513] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.308714 8072 leader_election.cc:290] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7013afe870964e1693fe739e7c7a4ad2 (127.2.36.65:33371), 1a58e5d41bbf48f1848bbbea7998b645 (127.2.36.66:46219)
I20250115 02:52:59.329897 8069 raft_consensus.cc:491] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:59.331939 8069 raft_consensus.cc:513] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.335979 8069 leader_election.cc:290] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a58e5d41bbf48f1848bbbea7998b645 (127.2.36.66:46219), 36b1e5772ad44039b76198414b571cf1 (127.2.36.67:33757)
I20250115 02:52:59.357983 8069 raft_consensus.cc:491] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:59.358718 8069 raft_consensus.cc:513] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.363752 8069 leader_election.cc:290] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a58e5d41bbf48f1848bbbea7998b645 (127.2.36.66:46219), 36b1e5772ad44039b76198414b571cf1 (127.2.36.67:33757)
I20250115 02:52:59.373396 8075 raft_consensus.cc:491] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:52:59.374204 8075 raft_consensus.cc:513] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.375593 7860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4118be3dcfc49bf8a7afe9210e91e9e" candidate_uuid: "36b1e5772ad44039b76198414b571cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7013afe870964e1693fe739e7c7a4ad2" is_pre_election: true
I20250115 02:52:59.376560 7860 raft_consensus.cc:2463] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b1e5772ad44039b76198414b571cf1 in term 0.
I20250115 02:52:59.379048 8075 leader_election.cc:290] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7013afe870964e1693fe739e7c7a4ad2 (127.2.36.65:33371), 36b1e5772ad44039b76198414b571cf1 (127.2.36.67:33757)
I20250115 02:52:59.380039 7860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e533b7b65683437aa22299a11437290f" candidate_uuid: "1a58e5d41bbf48f1848bbbea7998b645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7013afe870964e1693fe739e7c7a4ad2" is_pre_election: true
I20250115 02:52:59.380797 7860 raft_consensus.cc:2463] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a58e5d41bbf48f1848bbbea7998b645 in term 0.
I20250115 02:52:59.383072 7901 leader_election.cc:304] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a58e5d41bbf48f1848bbbea7998b645, 7013afe870964e1693fe739e7c7a4ad2; no voters:
I20250115 02:52:59.383878 8016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e533b7b65683437aa22299a11437290f" candidate_uuid: "1a58e5d41bbf48f1848bbbea7998b645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b1e5772ad44039b76198414b571cf1" is_pre_election: true
I20250115 02:52:59.384402 8075 raft_consensus.cc:2798] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:59.384691 8016 raft_consensus.cc:2463] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a58e5d41bbf48f1848bbbea7998b645 in term 0.
I20250115 02:52:59.384864 8075 raft_consensus.cc:491] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:59.385378 8075 raft_consensus.cc:3054] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:59.393898 7938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4118be3dcfc49bf8a7afe9210e91e9e" candidate_uuid: "36b1e5772ad44039b76198414b571cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a58e5d41bbf48f1848bbbea7998b645" is_pre_election: true
I20250115 02:52:59.395047 7938 raft_consensus.cc:2463] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b1e5772ad44039b76198414b571cf1 in term 0.
I20250115 02:52:59.397730 8075 raft_consensus.cc:513] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.399708 8075 leader_election.cc:290] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [CANDIDATE]: Term 1 election: Requested vote from peers 7013afe870964e1693fe739e7c7a4ad2 (127.2.36.65:33371), 36b1e5772ad44039b76198414b571cf1 (127.2.36.67:33757)
I20250115 02:52:59.400585 7860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e533b7b65683437aa22299a11437290f" candidate_uuid: "1a58e5d41bbf48f1848bbbea7998b645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7013afe870964e1693fe739e7c7a4ad2"
I20250115 02:52:59.401522 7860 raft_consensus.cc:3054] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:59.411521 7860 raft_consensus.cc:2463] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a58e5d41bbf48f1848bbbea7998b645 in term 1.
I20250115 02:52:59.413247 8016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4118be3dcfc49bf8a7afe9210e91e9e" candidate_uuid: "7013afe870964e1693fe739e7c7a4ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b1e5772ad44039b76198414b571cf1" is_pre_election: true
I20250115 02:52:59.413417 7901 leader_election.cc:304] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a58e5d41bbf48f1848bbbea7998b645, 7013afe870964e1693fe739e7c7a4ad2; no voters:
I20250115 02:52:59.413936 8016 raft_consensus.cc:2463] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7013afe870964e1693fe739e7c7a4ad2 in term 0.
I20250115 02:52:59.414707 8075 raft_consensus.cc:2798] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:59.415146 8015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e533b7b65683437aa22299a11437290f" candidate_uuid: "7013afe870964e1693fe739e7c7a4ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b1e5772ad44039b76198414b571cf1" is_pre_election: true
I20250115 02:52:59.415807 8015 raft_consensus.cc:2463] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7013afe870964e1693fe739e7c7a4ad2 in term 0.
I20250115 02:52:59.416168 8075 raft_consensus.cc:695] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 LEADER]: Becoming Leader. State: Replica: 1a58e5d41bbf48f1848bbbea7998b645, State: Running, Role: LEADER
I20250115 02:52:59.417251 8075 consensus_queue.cc:237] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.419486 7814 leader_election.cc:304] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b1e5772ad44039b76198414b571cf1, 7013afe870964e1693fe739e7c7a4ad2; no voters:
I20250115 02:52:59.420388 8069 raft_consensus.cc:2798] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:59.420755 8069 raft_consensus.cc:491] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:59.421061 8069 raft_consensus.cc:3054] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:59.424888 7979 leader_election.cc:304] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b1e5772ad44039b76198414b571cf1, 7013afe870964e1693fe739e7c7a4ad2; no voters:
I20250115 02:52:59.428795 7814 leader_election.cc:304] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b1e5772ad44039b76198414b571cf1, 7013afe870964e1693fe739e7c7a4ad2; no voters:
I20250115 02:52:59.430042 8072 raft_consensus.cc:2798] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:52:59.430449 8072 raft_consensus.cc:491] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:52:59.430781 8072 raft_consensus.cc:3054] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:59.432610 8069 raft_consensus.cc:513] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.434794 8069 leader_election.cc:290] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a58e5d41bbf48f1848bbbea7998b645 (127.2.36.66:46219), 36b1e5772ad44039b76198414b571cf1 (127.2.36.67:33757)
I20250115 02:52:59.435889 8099 raft_consensus.cc:2758] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250115 02:52:59.437364 8015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4118be3dcfc49bf8a7afe9210e91e9e" candidate_uuid: "7013afe870964e1693fe739e7c7a4ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b1e5772ad44039b76198414b571cf1"
I20250115 02:52:59.438620 8072 raft_consensus.cc:513] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.439945 8015 raft_consensus.cc:2388] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7013afe870964e1693fe739e7c7a4ad2 in current term 1: Already voted for candidate 36b1e5772ad44039b76198414b571cf1 in this term.
W20250115 02:52:59.443464 7979 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.36.65:33371, user_credentials={real_user=slave}} blocked reactor thread for 47639.9us
I20250115 02:52:59.445092 8015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e533b7b65683437aa22299a11437290f" candidate_uuid: "1a58e5d41bbf48f1848bbbea7998b645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b1e5772ad44039b76198414b571cf1"
I20250115 02:52:59.445762 8015 raft_consensus.cc:3054] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:59.449404 7938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4118be3dcfc49bf8a7afe9210e91e9e" candidate_uuid: "36b1e5772ad44039b76198414b571cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a58e5d41bbf48f1848bbbea7998b645"
I20250115 02:52:59.450071 7938 raft_consensus.cc:3054] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:52:59.453080 7937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4118be3dcfc49bf8a7afe9210e91e9e" candidate_uuid: "7013afe870964e1693fe739e7c7a4ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a58e5d41bbf48f1848bbbea7998b645" is_pre_election: true
I20250115 02:52:59.454576 7936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e533b7b65683437aa22299a11437290f" candidate_uuid: "7013afe870964e1693fe739e7c7a4ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a58e5d41bbf48f1848bbbea7998b645" is_pre_election: true
I20250115 02:52:59.456076 7935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4118be3dcfc49bf8a7afe9210e91e9e" candidate_uuid: "7013afe870964e1693fe739e7c7a4ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a58e5d41bbf48f1848bbbea7998b645"
I20250115 02:52:59.459039 8072 leader_election.cc:290] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [CANDIDATE]: Term 1 election: Requested vote from peers 7013afe870964e1693fe739e7c7a4ad2 (127.2.36.65:33371), 1a58e5d41bbf48f1848bbbea7998b645 (127.2.36.66:46219)
I20250115 02:52:59.460657 7860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4118be3dcfc49bf8a7afe9210e91e9e" candidate_uuid: "36b1e5772ad44039b76198414b571cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7013afe870964e1693fe739e7c7a4ad2"
I20250115 02:52:59.464628 7860 raft_consensus.cc:2388] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 36b1e5772ad44039b76198414b571cf1 in current term 1: Already voted for candidate 7013afe870964e1693fe739e7c7a4ad2 in this term.
I20250115 02:52:59.463647 7938 raft_consensus.cc:2463] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b1e5772ad44039b76198414b571cf1 in term 1.
I20250115 02:52:59.465229 7815 leader_election.cc:304] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7013afe870964e1693fe739e7c7a4ad2; no voters: 1a58e5d41bbf48f1848bbbea7998b645, 36b1e5772ad44039b76198414b571cf1
I20250115 02:52:59.466280 8099 raft_consensus.cc:2743] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250115 02:52:59.468062 7980 leader_election.cc:304] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a58e5d41bbf48f1848bbbea7998b645, 36b1e5772ad44039b76198414b571cf1; no voters: 7013afe870964e1693fe739e7c7a4ad2
I20250115 02:52:59.470005 8100 raft_consensus.cc:2798] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:52:59.470717 8100 raft_consensus.cc:695] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 1 LEADER]: Becoming Leader. State: Replica: 36b1e5772ad44039b76198414b571cf1, State: Running, Role: LEADER
I20250115 02:52:59.469998 8015 raft_consensus.cc:2463] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a58e5d41bbf48f1848bbbea7998b645 in term 1.
I20250115 02:52:59.471824 8100 consensus_queue.cc:237] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } }
I20250115 02:52:59.488875 7721 catalog_manager.cc:5526] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36b1e5772ad44039b76198414b571cf1 (127.2.36.67). New cstate: current_term: 1 leader_uuid: "36b1e5772ad44039b76198414b571cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } health_report { overall_health: HEALTHY } } }
I20250115 02:52:59.488869 7722 catalog_manager.cc:5526] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a58e5d41bbf48f1848bbbea7998b645 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "1a58e5d41bbf48f1848bbbea7998b645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 } health_report { overall_health: UNKNOWN } } }
I20250115 02:52:59.600225 8075 consensus_queue.cc:1035] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:59.643945 8080 consensus_queue.cc:1035] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250115 02:52:59.799883 8080 consensus_queue.cc:1035] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b1e5772ad44039b76198414b571cf1" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:59.876734 8100 consensus_queue.cc:1035] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:59.884204 8080 consensus_queue.cc:1035] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7013afe870964e1693fe739e7c7a4ad2" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:52:59.952852 8100 consensus_queue.cc:1035] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a58e5d41bbf48f1848bbbea7998b645" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:00.388402 2193 master.cc:537] Master@127.2.36.124:45399 shutting down...
I20250115 02:53:00.435808 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:00.436883 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:00.437462 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P a77a4fffef954686be239477d1274168: stopping tablet replica
W20250115 02:53:00.448220 2193 connection.cc:441] server connection from 127.0.0.1:49934 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49934 (request call id 105) could send its response
I20250115 02:53:00.483697 2193 master.cc:559] Master@127.2.36.124:45399 shutdown complete.
I20250115 02:53:01.587929 8146 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a77a4fffef954686be239477d1274168)
I20250115 02:53:01.588362 8146 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
I20250115 02:53:01.590014 8146 leader_election.cc:290] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 86b72de984ad4484a7f26a5d0377e575 (127.2.36.125:41901), a77a4fffef954686be239477d1274168 (127.2.36.124:45399)
I20250115 02:53:01.591078 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87f997285c2c42d48e14a596166dcb4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "86b72de984ad4484a7f26a5d0377e575" is_pre_election: true
W20250115 02:53:01.598331 7576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45399: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20250115 02:53:01.603559 7576 leader_election.cc:336] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a77a4fffef954686be239477d1274168 (127.2.36.124:45399): Network error: Client connection negotiation failed: client connection to 127.2.36.124:45399: connect: Connection refused (error 111)
I20250115 02:53:01.604048 7576 leader_election.cc:304] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87f997285c2c42d48e14a596166dcb4b; no voters: 86b72de984ad4484a7f26a5d0377e575, a77a4fffef954686be239477d1274168
I20250115 02:53:01.604799 8146 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 02:53:01.847432 8149 raft_consensus.cc:491] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a77a4fffef954686be239477d1274168)
I20250115 02:53:01.848603 8149 raft_consensus.cc:513] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
I20250115 02:53:01.852594 7602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86b72de984ad4484a7f26a5d0377e575" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "87f997285c2c42d48e14a596166dcb4b" is_pre_election: true
I20250115 02:53:01.853271 7602 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b72de984ad4484a7f26a5d0377e575 in term 1.
I20250115 02:53:01.854936 7643 leader_election.cc:304] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b72de984ad4484a7f26a5d0377e575, 87f997285c2c42d48e14a596166dcb4b; no voters:
I20250115 02:53:01.853458 8149 leader_election.cc:290] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87f997285c2c42d48e14a596166dcb4b (127.2.36.126:38529), a77a4fffef954686be239477d1274168 (127.2.36.124:45399)
I20250115 02:53:01.858250 8149 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:53:01.860685 8149 raft_consensus.cc:491] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a77a4fffef954686be239477d1274168)
I20250115 02:53:01.861253 8149 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 1 FOLLOWER]: Advancing to term 2
W20250115 02:53:01.874976 7641 leader_election.cc:336] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a77a4fffef954686be239477d1274168 (127.2.36.124:45399): Network error: Client connection negotiation failed: client connection to 127.2.36.124:45399: connect: Connection refused (error 111)
I20250115 02:53:01.874190 8149 raft_consensus.cc:513] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
I20250115 02:53:01.881110 8149 leader_election.cc:290] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [CANDIDATE]: Term 2 election: Requested vote from peers 87f997285c2c42d48e14a596166dcb4b (127.2.36.126:38529), a77a4fffef954686be239477d1274168 (127.2.36.124:45399)
I20250115 02:53:01.881685 7602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86b72de984ad4484a7f26a5d0377e575" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "87f997285c2c42d48e14a596166dcb4b"
I20250115 02:53:01.882328 7602 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 1 FOLLOWER]: Advancing to term 2
W20250115 02:53:01.893580 7641 leader_election.cc:336] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a77a4fffef954686be239477d1274168 (127.2.36.124:45399): Network error: Client connection negotiation failed: client connection to 127.2.36.124:45399: connect: Connection refused (error 111)
I20250115 02:53:01.894187 7602 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b72de984ad4484a7f26a5d0377e575 in term 2.
I20250115 02:53:01.896135 7643 leader_election.cc:304] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 86b72de984ad4484a7f26a5d0377e575, 87f997285c2c42d48e14a596166dcb4b; no voters: a77a4fffef954686be239477d1274168
I20250115 02:53:01.898376 8149 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 2 FOLLOWER]: Leader election won for term 2
I20250115 02:53:01.898948 8149 raft_consensus.cc:695] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 2 LEADER]: Becoming Leader. State: Replica: 86b72de984ad4484a7f26a5d0377e575, State: Running, Role: LEADER
I20250115 02:53:01.902057 8149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } }
I20250115 02:53:01.911070 8150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86b72de984ad4484a7f26a5d0377e575. Latest consensus state: current_term: 2 leader_uuid: "86b72de984ad4484a7f26a5d0377e575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:53:01.911948 8150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:01.919775 8155 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:01.924773 8155 catalog_manager.cc:670] Loaded metadata for table ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51]
I20250115 02:53:01.933624 8155 tablet_loader.cc:96] loaded metadata for tablet e4118be3dcfc49bf8a7afe9210e91e9e (table ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51])
I20250115 02:53:01.935226 8155 tablet_loader.cc:96] loaded metadata for tablet e533b7b65683437aa22299a11437290f (table ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51])
I20250115 02:53:01.936475 8155 tablet_loader.cc:96] loaded metadata for tablet efe29696b7a4432a918ed169149c1c41 (table ksck-test-table [id=0aa0dbfe200245b7a84e1e01a9f24f51])
I20250115 02:53:01.937589 8155 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:01.941685 8155 catalog_manager.cc:1260] Loaded cluster ID: 388c38c4263e44329a79a6b325b50d71
I20250115 02:53:01.942060 8155 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:01.947654 8155 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:01.952476 8155 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575: Loaded TSK: 0
I20250115 02:53:01.953223 8155 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:01.976068 8042 heartbeater.cc:501] Master 127.2.36.125:41901 was elected leader, sending a full tablet report...
I20250115 02:53:01.976948 7964 heartbeater.cc:501] Master 127.2.36.125:41901 was elected leader, sending a full tablet report...
I20250115 02:53:01.979694 7886 heartbeater.cc:501] Master 127.2.36.125:41901 was elected leader, sending a full tablet report...
I20250115 02:53:02.083943 7926 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250115 02:53:02.087895 8004 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:53:02.093466 7849 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250115 02:53:02.372879 7641 consensus_peers.cc:487] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 -> Peer a77a4fffef954686be239477d1274168 (127.2.36.124:45399): Couldn't send request to peer a77a4fffef954686be239477d1274168. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:53:02.469308 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:53:02.516341 7602 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 2 FOLLOWER]: Refusing update from remote peer 86b72de984ad4484a7f26a5d0377e575: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250115 02:53:02.517750 8150 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250115 02:53:02.525825 8167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86b72de984ad4484a7f26a5d0377e575. Latest consensus state: current_term: 2 leader_uuid: "86b72de984ad4484a7f26a5d0377e575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:53:02.526618 8167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:02.528658 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:02.529740 2193 tablet_replica.cc:331] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2: stopping tablet replica
I20250115 02:53:02.530550 2193 raft_consensus.cc:2238] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:02.531085 2193 raft_consensus.cc:2267] T e4118be3dcfc49bf8a7afe9210e91e9e P 7013afe870964e1693fe739e7c7a4ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:02.533645 2193 tablet_replica.cc:331] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2: stopping tablet replica
I20250115 02:53:02.534317 2193 raft_consensus.cc:2238] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:02.533442 8169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "86b72de984ad4484a7f26a5d0377e575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:53:02.535506 8169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:02.534667 2193 raft_consensus.cc:2267] T e533b7b65683437aa22299a11437290f P 7013afe870964e1693fe739e7c7a4ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:02.540562 2193 tablet_replica.cc:331] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2: stopping tablet replica
I20250115 02:53:02.540377 8167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "86b72de984ad4484a7f26a5d0377e575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f997285c2c42d48e14a596166dcb4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 38529 } } peers { permanent_uuid: "86b72de984ad4484a7f26a5d0377e575" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 41901 } } peers { permanent_uuid: "a77a4fffef954686be239477d1274168" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45399 } } }
I20250115 02:53:02.541174 8167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:02.542057 2193 raft_consensus.cc:2238] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:02.542511 2193 raft_consensus.cc:2267] T efe29696b7a4432a918ed169149c1c41 P 7013afe870964e1693fe739e7c7a4ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:02.567234 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
W20250115 02:53:02.588299 7979 consensus_peers.cc:487] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 -> Peer 7013afe870964e1693fe739e7c7a4ad2 (127.2.36.65:33371): Couldn't send request to peer 7013afe870964e1693fe739e7c7a4ad2. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:33371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:53:02.592864 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:53:02.638885 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:02.639607 2193 tablet_replica.cc:331] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645: stopping tablet replica
I20250115 02:53:02.640293 2193 raft_consensus.cc:2238] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:02.641238 2193 raft_consensus.cc:2267] T e533b7b65683437aa22299a11437290f P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:02.643240 2193 tablet_replica.cc:331] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645: stopping tablet replica
I20250115 02:53:02.643702 2193 raft_consensus.cc:2238] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:02.644045 2193 raft_consensus.cc:2267] T e4118be3dcfc49bf8a7afe9210e91e9e P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:02.645730 2193 tablet_replica.cc:331] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645: stopping tablet replica
I20250115 02:53:02.646127 2193 raft_consensus.cc:2238] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:02.646837 2193 raft_consensus.cc:2267] T efe29696b7a4432a918ed169149c1c41 P 1a58e5d41bbf48f1848bbbea7998b645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:02.669215 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:53:02.684273 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:53:02.714089 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:02.714839 2193 tablet_replica.cc:331] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1: stopping tablet replica
I20250115 02:53:02.715426 2193 raft_consensus.cc:2238] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:02.715853 2193 raft_consensus.cc:2267] T efe29696b7a4432a918ed169149c1c41 P 36b1e5772ad44039b76198414b571cf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:02.717829 2193 tablet_replica.cc:331] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1: stopping tablet replica
I20250115 02:53:02.718305 2193 raft_consensus.cc:2238] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:02.718693 2193 raft_consensus.cc:2267] T e533b7b65683437aa22299a11437290f P 36b1e5772ad44039b76198414b571cf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:02.720625 2193 tablet_replica.cc:331] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1: stopping tablet replica
I20250115 02:53:02.721047 2193 raft_consensus.cc:2238] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:02.721771 2193 raft_consensus.cc:2267] T e4118be3dcfc49bf8a7afe9210e91e9e P 36b1e5772ad44039b76198414b571cf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:02.742616 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:53:02.758518 2193 master.cc:537] Master@127.2.36.126:38529 shutting down...
I20250115 02:53:02.778818 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:02.779371 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:02.779664 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 87f997285c2c42d48e14a596166dcb4b: stopping tablet replica
I20250115 02:53:02.799088 2193 master.cc:559] Master@127.2.36.126:38529 shutdown complete.
I20250115 02:53:02.813010 2193 master.cc:537] Master@127.2.36.125:41901 shutting down...
I20250115 02:53:02.834510 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 2 LEADER]: Raft consensus shutting down.
I20250115 02:53:02.835469 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:02.835879 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 86b72de984ad4484a7f26a5d0377e575: stopping tablet replica
I20250115 02:53:02.856602 2193 master.cc:559] Master@127.2.36.125:41901 shutdown complete.
[ OK ] RemoteKsckTest.TestLeaderMasterDown (5192 ms)
[ RUN ] RemoteKsckTest.TestClusterWithLocation
I20250115 02:53:02.889561 2193 test_util.cc:274] Using random seed: -961416635
I20250115 02:53:02.890491 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:43155,127.2.36.125:34261,127.2.36.124:39147
I20250115 02:53:02.891767 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:02.897747 8175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:02.898792 8176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:02.899482 8178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:02.900175 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:02.901079 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:02.901270 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:02.901443 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909582901426 us; error 0 us; skew 500 ppm
I20250115 02:53:02.902087 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:02.904847 2193 webserver.cc:458] Webserver started at http://127.2.36.126:39643/ using document root <none> and password file <none>
I20250115 02:53:02.905324 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:02.905493 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:02.905702 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:02.906837 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "412cd59ec6454237b54608b8639fd2cb"
format_stamp: "Formatted at 2025-01-15 02:53:02 on dist-test-slave-1lv2"
I20250115 02:53:02.911141 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20250115 02:53:02.914399 8183 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:02.915141 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:53:02.915411 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "412cd59ec6454237b54608b8639fd2cb"
format_stamp: "Formatted at 2025-01-15 02:53:02 on dist-test-slave-1lv2"
I20250115 02:53:02.915678 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:02.924808 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:02.925937 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:02.962415 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:43155
I20250115 02:53:02.962517 8234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:43155 every 8 connection(s)
I20250115 02:53:02.966045 8235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:02.966325 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:02.972543 8237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:02.973131 8238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:02.973620 8235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:02.975154 8240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:02.977036 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:02.978945 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:02.979171 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:02.979324 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909582979305 us; error 0 us; skew 500 ppm
I20250115 02:53:02.979900 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:02.982684 2193 webserver.cc:458] Webserver started at http://127.2.36.125:43203/ using document root <none> and password file <none>
I20250115 02:53:02.983184 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:02.983369 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:02.983673 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:02.984766 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "e8b6772d16f44e0f9ad5d9936490d280"
format_stamp: "Formatted at 2025-01-15 02:53:02 on dist-test-slave-1lv2"
I20250115 02:53:02.987092 8235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:02.989768 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
W20250115 02:53:02.992116 8235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:34261: Network error: Client connection negotiation failed: client connection to 127.2.36.125:34261: connect: Connection refused (error 111)
I20250115 02:53:02.993806 8248 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:02.994596 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:02.994948 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "e8b6772d16f44e0f9ad5d9936490d280"
format_stamp: "Formatted at 2025-01-15 02:53:02 on dist-test-slave-1lv2"
I20250115 02:53:02.995245 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:03.010571 8235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } attempt: 1
I20250115 02:53:03.012694 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:03.014174 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250115 02:53:03.014747 8235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:34261: Network error: Client connection negotiation failed: client connection to 127.2.36.125:34261: connect: Connection refused (error 111)
I20250115 02:53:03.048466 8235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } attempt: 2
W20250115 02:53:03.052951 8235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:34261: Network error: Client connection negotiation failed: client connection to 127.2.36.125:34261: connect: Connection refused (error 111)
I20250115 02:53:03.054169 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:34261
I20250115 02:53:03.054258 8299 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:34261 every 8 connection(s)
I20250115 02:53:03.057785 8300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:03.057948 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:03.064292 8302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:03.064962 8300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:03.067108 8303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:03.068619 8305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:03.069370 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:03.070271 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:03.070477 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:03.070643 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909583070625 us; error 0 us; skew 500 ppm
I20250115 02:53:03.071156 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:03.074141 2193 webserver.cc:458] Webserver started at http://127.2.36.124:32835/ using document root <none> and password file <none>
I20250115 02:53:03.074654 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:03.074882 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:03.075187 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:03.076437 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "6352b3d955204093b104d1efc24ebc88"
format_stamp: "Formatted at 2025-01-15 02:53:03 on dist-test-slave-1lv2"
I20250115 02:53:03.077090 8300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:03.081394 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20250115 02:53:03.085709 8312 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:03.087303 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250115 02:53:03.087673 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "6352b3d955204093b104d1efc24ebc88"
format_stamp: "Formatted at 2025-01-15 02:53:03 on dist-test-slave-1lv2"
I20250115 02:53:03.088012 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:03.091660 8300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:03.097973 8300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:39147: Network error: Client connection negotiation failed: client connection to 127.2.36.124:39147: connect: Connection refused (error 111)
I20250115 02:53:03.109102 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:03.110348 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:03.130546 8300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } attempt: 1
W20250115 02:53:03.135671 8300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:39147: Network error: Client connection negotiation failed: client connection to 127.2.36.124:39147: connect: Connection refused (error 111)
I20250115 02:53:03.147380 8235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } attempt: 3
I20250115 02:53:03.149719 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:39147
I20250115 02:53:03.149794 8364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:39147 every 8 connection(s)
I20250115 02:53:03.153534 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:53:03.154536 8365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:03.157815 8235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:03.161751 8365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:03.172619 8365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:03.175257 8235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: Bootstrap starting.
I20250115 02:53:03.181270 8235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:03.182577 8365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:03.186137 8235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: No bootstrap required, opened a new log
I20250115 02:53:03.188759 8235 raft_consensus.cc:357] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:03.189435 8235 raft_consensus.cc:383] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:03.189966 8235 raft_consensus.cc:738] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412cd59ec6454237b54608b8639fd2cb, State: Initialized, Role: FOLLOWER
I20250115 02:53:03.190646 8235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:03.192842 8370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:03.193658 8370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:03.194415 8235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:03.198211 8365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: Bootstrap starting.
I20250115 02:53:03.203069 8365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:03.206373 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:53:03.207652 8365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: No bootstrap required, opened a new log
W20250115 02:53:03.209254 8382 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:03.209543 8382 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:03.209847 8365 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:03.210400 8365 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:03.210621 8365 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6352b3d955204093b104d1efc24ebc88, State: Initialized, Role: FOLLOWER
I20250115 02:53:03.211217 8365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:03.212771 8383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:03.213337 8383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:03.214073 8365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:03.215186 8300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } attempt: 2
W20250115 02:53:03.229185 8394 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:03.229488 8394 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:03.231564 8300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: Bootstrap starting.
I20250115 02:53:03.236279 8300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:03.240351 8300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: No bootstrap required, opened a new log
I20250115 02:53:03.242372 8300 raft_consensus.cc:357] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:03.242969 8300 raft_consensus.cc:383] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:03.243178 8300 raft_consensus.cc:738] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b6772d16f44e0f9ad5d9936490d280, State: Initialized, Role: FOLLOWER
I20250115 02:53:03.243738 8300 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:03.245399 8396 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:03.245957 8396 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:03.246567 8300 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:03.255123 8370 raft_consensus.cc:491] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:03.255702 8370 raft_consensus.cc:513] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:03.256822 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:53:03.257755 8370 leader_election.cc:290] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8b6772d16f44e0f9ad5d9936490d280 (127.2.36.125:34261), 6352b3d955204093b104d1efc24ebc88 (127.2.36.124:39147)
I20250115 02:53:03.258682 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:03.258993 8275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "412cd59ec6454237b54608b8639fd2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6772d16f44e0f9ad5d9936490d280" is_pre_election: true
I20250115 02:53:03.258988 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "412cd59ec6454237b54608b8639fd2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6352b3d955204093b104d1efc24ebc88" is_pre_election: true
I20250115 02:53:03.259825 8275 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412cd59ec6454237b54608b8639fd2cb in term 0.
I20250115 02:53:03.259907 8340 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412cd59ec6454237b54608b8639fd2cb in term 0.
I20250115 02:53:03.261096 8187 leader_election.cc:304] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412cd59ec6454237b54608b8639fd2cb, 6352b3d955204093b104d1efc24ebc88; no voters:
I20250115 02:53:03.261899 8370 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250115 02:53:03.262126 8407 catalog_manager.cc:1559] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250115 02:53:03.262216 8370 raft_consensus.cc:491] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250115 02:53:03.262562 8407 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:03.262636 8370 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 0 FOLLOWER]: Advancing to term 1
W20250115 02:53:03.267369 8408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:03.268023 8409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:03.269448 8411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:03.270287 8370 raft_consensus.cc:513] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:03.271150 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:03.272159 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250115 02:53:03.272197 8370 leader_election.cc:290] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [CANDIDATE]: Term 1 election: Requested vote from peers e8b6772d16f44e0f9ad5d9936490d280 (127.2.36.125:34261), 6352b3d955204093b104d1efc24ebc88 (127.2.36.124:39147)
W20250115 02:53:03.272604 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:03.272895 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909583272874 us; error 0 us; skew 500 ppm
I20250115 02:53:03.273221 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "412cd59ec6454237b54608b8639fd2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6352b3d955204093b104d1efc24ebc88"
I20250115 02:53:03.273195 8275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "412cd59ec6454237b54608b8639fd2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6772d16f44e0f9ad5d9936490d280"
I20250115 02:53:03.273734 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:03.273954 8340 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:03.274029 8275 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:03.280447 2193 webserver.cc:458] Webserver started at http://127.2.36.65:42317/ using document root <none> and password file <none>
I20250115 02:53:03.280680 8340 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412cd59ec6454237b54608b8639fd2cb in term 1.
I20250115 02:53:03.281239 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:03.281589 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:03.281751 8187 leader_election.cc:304] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412cd59ec6454237b54608b8639fd2cb, 6352b3d955204093b104d1efc24ebc88; no voters:
I20250115 02:53:03.281922 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:03.282230 8275 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412cd59ec6454237b54608b8639fd2cb in term 1.
I20250115 02:53:03.282506 8370 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:03.283469 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94"
format_stamp: "Formatted at 2025-01-15 02:53:03 on dist-test-slave-1lv2"
I20250115 02:53:03.283612 8370 raft_consensus.cc:695] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 1 LEADER]: Becoming Leader. State: Replica: 412cd59ec6454237b54608b8639fd2cb, State: Running, Role: LEADER
I20250115 02:53:03.284406 8370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:03.288069 8415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412cd59ec6454237b54608b8639fd2cb. Latest consensus state: current_term: 1 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:03.288938 8415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:03.289777 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250115 02:53:03.290386 8418 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:03.293551 8419 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:03.294428 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250115 02:53:03.294739 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94"
format_stamp: "Formatted at 2025-01-15 02:53:03 on dist-test-slave-1lv2"
I20250115 02:53:03.295082 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:03.296069 8418 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:03.304802 8340 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 1 FOLLOWER]: Refusing update from remote peer 412cd59ec6454237b54608b8639fd2cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:03.305687 8275 raft_consensus.cc:1270] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 1 FOLLOWER]: Refusing update from remote peer 412cd59ec6454237b54608b8639fd2cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:03.306069 8415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:03.307163 8370 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:03.308290 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:03.309710 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:03.313032 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:03.315543 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:03.315742 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:03.315974 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:03.316125 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:03.326467 8383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412cd59ec6454237b54608b8639fd2cb. Latest consensus state: current_term: 1 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:03.327293 8383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:03.334048 8396 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412cd59ec6454237b54608b8639fd2cb. Latest consensus state: current_term: 1 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:03.335342 8396 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:03.341691 8383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:03.342594 8383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:03.342818 8396 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:03.343575 8396 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:03.348681 8418 catalog_manager.cc:1348] Generated new cluster ID: e2d47ad2480e4ddaa1f0b56f7191cea2
I20250115 02:53:03.349033 8418 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:03.347919 8370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:03.349375 8370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:03.349938 8370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:03.350633 8370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:03.375134 8418 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:53:03.376824 8418 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:03.399348 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:36051
I20250115 02:53:03.399435 8491 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:36051 every 8 connection(s)
I20250115 02:53:03.419435 8418 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: Generated new TSK 0
I20250115 02:53:03.420234 8418 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:03.420114 8492 heartbeater.cc:346] Connected to a master server at 127.2.36.125:34261
I20250115 02:53:03.420941 8492 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:03.421837 8492 heartbeater.cc:509] Master 127.2.36.125:34261 requested a full tablet report, sending...
I20250115 02:53:03.423264 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:03.424988 8265 ts_manager.cc:194] Registered new tserver with Master: 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65:36051)
W20250115 02:53:03.440491 8499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:03.441134 8500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:03.442303 8493 heartbeater.cc:346] Connected to a master server at 127.2.36.124:39147
I20250115 02:53:03.442873 8493 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:03.443822 8493 heartbeater.cc:509] Master 127.2.36.124:39147 requested a full tablet report, sending...
I20250115 02:53:03.445312 8495 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43155
I20250115 02:53:03.445746 8495 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:03.446637 8495 heartbeater.cc:509] Master 127.2.36.126:43155 requested a full tablet report, sending...
I20250115 02:53:03.449126 8200 ts_manager.cc:194] Registered new tserver with Master: 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65:36051)
I20250115 02:53:03.449433 8330 ts_manager.cc:194] Registered new tserver with Master: 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65:36051)
W20250115 02:53:03.451241 8502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:03.451339 8200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55724
I20250115 02:53:03.451822 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:03.452839 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:03.453039 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:03.453234 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909583453215 us; error 0 us; skew 500 ppm
I20250115 02:53:03.453903 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:03.456557 2193 webserver.cc:458] Webserver started at http://127.2.36.66:38409/ using document root <none> and password file <none>
I20250115 02:53:03.456971 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:03.457118 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:03.457314 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:03.458267 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "c04b74f0e5444b2f94c83d82c89c8426"
format_stamp: "Formatted at 2025-01-15 02:53:03 on dist-test-slave-1lv2"
I20250115 02:53:03.462853 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250115 02:53:03.464402 8495 heartbeater.cc:501] Master 127.2.36.126:43155 was elected leader, sending a full tablet report...
I20250115 02:53:03.466580 8507 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:03.468832 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250115 02:53:03.469125 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "c04b74f0e5444b2f94c83d82c89c8426"
format_stamp: "Formatted at 2025-01-15 02:53:03 on dist-test-slave-1lv2"
I20250115 02:53:03.469405 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:03.495658 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:03.496901 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:03.498145 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:03.500600 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:03.500898 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250115 02:53:03.501168 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:03.501377 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:03.548286 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:46713
I20250115 02:53:03.548385 8569 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:46713 every 8 connection(s)
I20250115 02:53:03.578198 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:03.578641 8570 heartbeater.cc:346] Connected to a master server at 127.2.36.125:34261
I20250115 02:53:03.579030 8570 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:03.579818 8570 heartbeater.cc:509] Master 127.2.36.125:34261 requested a full tablet report, sending...
I20250115 02:53:03.582845 8572 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43155
I20250115 02:53:03.583130 8265 ts_manager.cc:194] Registered new tserver with Master: c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713)
I20250115 02:53:03.583256 8572 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:03.584381 8572 heartbeater.cc:509] Master 127.2.36.126:43155 requested a full tablet report, sending...
I20250115 02:53:03.586786 8200 ts_manager.cc:194] Registered new tserver with Master: c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713)
W20250115 02:53:03.588764 8578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:03.591995 8571 heartbeater.cc:346] Connected to a master server at 127.2.36.124:39147
I20250115 02:53:03.592420 8571 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:03.588855 8200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55740
I20250115 02:53:03.593313 8571 heartbeater.cc:509] Master 127.2.36.124:39147 requested a full tablet report, sending...
I20250115 02:53:03.596077 8330 ts_manager.cc:194] Registered new tserver with Master: c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713)
W20250115 02:53:03.598296 8580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:03.599388 8577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:03.601197 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:03.602036 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:03.602246 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:03.602417 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909583602404 us; error 0 us; skew 500 ppm
I20250115 02:53:03.603082 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:03.605746 8572 heartbeater.cc:501] Master 127.2.36.126:43155 was elected leader, sending a full tablet report...
I20250115 02:53:03.606065 2193 webserver.cc:458] Webserver started at http://127.2.36.67:46403/ using document root <none> and password file <none>
I20250115 02:53:03.606736 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:03.606984 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:03.607275 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:03.608667 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "6e224d15a8964126b95c016aa46efb28"
format_stamp: "Formatted at 2025-01-15 02:53:03 on dist-test-slave-1lv2"
I20250115 02:53:03.614055 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250115 02:53:03.618221 8585 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:03.619136 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250115 02:53:03.619531 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "6e224d15a8964126b95c016aa46efb28"
format_stamp: "Formatted at 2025-01-15 02:53:03 on dist-test-slave-1lv2"
I20250115 02:53:03.619932 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:03.632509 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:03.634148 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:03.635905 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:03.639111 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:03.639412 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:03.639659 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:03.639811 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:03.698997 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:38517
I20250115 02:53:03.699098 8647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:38517 every 8 connection(s)
I20250115 02:53:03.732883 8648 heartbeater.cc:346] Connected to a master server at 127.2.36.125:34261
I20250115 02:53:03.733371 8648 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:03.734246 8648 heartbeater.cc:509] Master 127.2.36.125:34261 requested a full tablet report, sending...
I20250115 02:53:03.738890 8264 ts_manager.cc:194] Registered new tserver with Master: 6e224d15a8964126b95c016aa46efb28 (127.2.36.67:38517)
I20250115 02:53:03.744076 8650 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43155
I20250115 02:53:03.744424 8650 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:03.745110 8650 heartbeater.cc:509] Master 127.2.36.126:43155 requested a full tablet report, sending...
I20250115 02:53:03.748270 8200 ts_manager.cc:194] Registered new tserver with Master: 6e224d15a8964126b95c016aa46efb28 (127.2.36.67:38517)
I20250115 02:53:03.748953 8649 heartbeater.cc:346] Connected to a master server at 127.2.36.124:39147
I20250115 02:53:03.749293 8649 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:03.750038 8649 heartbeater.cc:509] Master 127.2.36.124:39147 requested a full tablet report, sending...
I20250115 02:53:03.750031 8200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55756
I20250115 02:53:03.754321 8329 ts_manager.cc:194] Registered new tserver with Master: 6e224d15a8964126b95c016aa46efb28 (127.2.36.67:38517)
I20250115 02:53:03.755139 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.027551046s
I20250115 02:53:03.766795 8650 heartbeater.cc:501] Master 127.2.36.126:43155 was elected leader, sending a full tablet report...
I20250115 02:53:03.800861 8200 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55762:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:53:03.806311 8200 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:53:03.926688 8534 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb519c79297400785d3ce4bb2fc232e (DEFAULT_TABLE table=ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:03.928076 8534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb519c79297400785d3ce4bb2fc232e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:03.930718 8533 tablet_service.cc:1467] Processing CreateTablet for tablet f2c0a914505c44b59cc22b05ed63518a (DEFAULT_TABLE table=ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:03.931916 8533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2c0a914505c44b59cc22b05ed63518a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:03.926872 8535 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2e395c8a0149ae9c88f74272f7b8df (DEFAULT_TABLE table=ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:03.935557 8613 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2e395c8a0149ae9c88f74272f7b8df (DEFAULT_TABLE table=ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:03.936158 8535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e2e395c8a0149ae9c88f74272f7b8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:03.937471 8613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e2e395c8a0149ae9c88f74272f7b8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:03.938565 8610 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb519c79297400785d3ce4bb2fc232e (DEFAULT_TABLE table=ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:03.940712 8612 tablet_service.cc:1467] Processing CreateTablet for tablet f2c0a914505c44b59cc22b05ed63518a (DEFAULT_TABLE table=ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:03.940621 8610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb519c79297400785d3ce4bb2fc232e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:03.942741 8612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2c0a914505c44b59cc22b05ed63518a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:03.949298 8456 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2e395c8a0149ae9c88f74272f7b8df (DEFAULT_TABLE table=ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:03.951752 8455 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb519c79297400785d3ce4bb2fc232e (DEFAULT_TABLE table=ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:03.954038 8454 tablet_service.cc:1467] Processing CreateTablet for tablet f2c0a914505c44b59cc22b05ed63518a (DEFAULT_TABLE table=ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:03.950619 8456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e2e395c8a0149ae9c88f74272f7b8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:03.958531 8455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb519c79297400785d3ce4bb2fc232e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:03.960004 8454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2c0a914505c44b59cc22b05ed63518a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:04.027994 8670 tablet_bootstrap.cc:492] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426: Bootstrap starting.
I20250115 02:53:04.044636 8670 tablet_bootstrap.cc:654] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:04.059978 8669 tablet_bootstrap.cc:492] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28: Bootstrap starting.
I20250115 02:53:04.073087 8670 tablet_bootstrap.cc:492] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426: No bootstrap required, opened a new log
I20250115 02:53:04.073763 8670 ts_tablet_manager.cc:1397] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.006s
I20250115 02:53:04.079463 8670 raft_consensus.cc:357] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.080207 8670 raft_consensus.cc:383] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:04.081094 8670 raft_consensus.cc:738] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04b74f0e5444b2f94c83d82c89c8426, State: Initialized, Role: FOLLOWER
I20250115 02:53:04.081833 8670 consensus_queue.cc:260] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.098803 8669 tablet_bootstrap.cc:654] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:04.113647 8670 ts_tablet_manager.cc:1428] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426: Time spent starting tablet: real 0.039s user 0.010s sys 0.025s
I20250115 02:53:04.114846 8670 tablet_bootstrap.cc:492] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426: Bootstrap starting.
I20250115 02:53:04.123456 8670 tablet_bootstrap.cc:654] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:04.128247 8669 tablet_bootstrap.cc:492] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28: No bootstrap required, opened a new log
I20250115 02:53:04.128765 8669 ts_tablet_manager.cc:1397] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.003s
I20250115 02:53:04.132047 8673 tablet_bootstrap.cc:492] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94: Bootstrap starting.
I20250115 02:53:04.134001 8669 raft_consensus.cc:357] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.134784 8669 raft_consensus.cc:383] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:04.141031 8669 raft_consensus.cc:738] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e224d15a8964126b95c016aa46efb28, State: Initialized, Role: FOLLOWER
I20250115 02:53:04.141749 8669 consensus_queue.cc:260] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.146564 8673 tablet_bootstrap.cc:654] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:04.153613 8670 tablet_bootstrap.cc:492] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426: No bootstrap required, opened a new log
I20250115 02:53:04.154402 8670 ts_tablet_manager.cc:1397] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.018s
I20250115 02:53:04.155834 8676 raft_consensus.cc:491] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:04.156361 8676 raft_consensus.cc:513] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.157723 8669 ts_tablet_manager.cc:1428] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28: Time spent starting tablet: real 0.029s user 0.011s sys 0.001s
I20250115 02:53:04.167949 8669 tablet_bootstrap.cc:492] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28: Bootstrap starting.
I20250115 02:53:04.157343 8670 raft_consensus.cc:357] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.173161 8670 raft_consensus.cc:383] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:04.173593 8670 raft_consensus.cc:738] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04b74f0e5444b2f94c83d82c89c8426, State: Initialized, Role: FOLLOWER
I20250115 02:53:04.174378 8670 consensus_queue.cc:260] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.177150 8670 ts_tablet_manager.cc:1428] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20250115 02:53:04.178267 8670 tablet_bootstrap.cc:492] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426: Bootstrap starting.
I20250115 02:53:04.178889 8545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2e395c8a0149ae9c88f74272f7b8df" candidate_uuid: "6e224d15a8964126b95c016aa46efb28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04b74f0e5444b2f94c83d82c89c8426" is_pre_election: true
I20250115 02:53:04.179654 8545 raft_consensus.cc:2463] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e224d15a8964126b95c016aa46efb28 in term 0.
I20250115 02:53:04.180888 8586 leader_election.cc:304] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e224d15a8964126b95c016aa46efb28, c04b74f0e5444b2f94c83d82c89c8426; no voters:
I20250115 02:53:04.183043 8677 raft_consensus.cc:2798] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:04.183418 8677 raft_consensus.cc:491] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:04.183763 8677 raft_consensus.cc:3054] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:04.184947 8670 tablet_bootstrap.cc:654] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:04.186645 8673 tablet_bootstrap.cc:492] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94: No bootstrap required, opened a new log
I20250115 02:53:04.187181 8673 ts_tablet_manager.cc:1397] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.004s
I20250115 02:53:04.190198 8673 raft_consensus.cc:357] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.191052 8673 raft_consensus.cc:383] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:04.195906 8673 raft_consensus.cc:738] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2b237918cd4b8c9cb8ff9cf1356f94, State: Initialized, Role: FOLLOWER
I20250115 02:53:04.197084 8670 tablet_bootstrap.cc:492] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426: No bootstrap required, opened a new log
I20250115 02:53:04.197686 8670 ts_tablet_manager.cc:1397] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.013s
I20250115 02:53:04.196660 8673 consensus_queue.cc:260] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.200565 8467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2e395c8a0149ae9c88f74272f7b8df" candidate_uuid: "6e224d15a8964126b95c016aa46efb28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" is_pre_election: true
I20250115 02:53:04.201669 8670 raft_consensus.cc:357] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
W20250115 02:53:04.202263 8588 leader_election.cc:343] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65:36051): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:04.202469 8669 tablet_bootstrap.cc:654] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:04.202495 8670 raft_consensus.cc:383] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:04.204651 8670 raft_consensus.cc:738] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04b74f0e5444b2f94c83d82c89c8426, State: Initialized, Role: FOLLOWER
I20250115 02:53:04.205361 8676 leader_election.cc:290] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713), 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65:36051)
I20250115 02:53:04.210242 8670 consensus_queue.cc:260] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.215130 8670 ts_tablet_manager.cc:1428] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20250115 02:53:04.225289 8669 tablet_bootstrap.cc:492] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28: No bootstrap required, opened a new log
I20250115 02:53:04.225819 8669 ts_tablet_manager.cc:1397] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.005s
I20250115 02:53:04.226413 8677 raft_consensus.cc:513] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.222302 8680 raft_consensus.cc:491] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:04.222890 8673 ts_tablet_manager.cc:1428] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94: Time spent starting tablet: real 0.035s user 0.002s sys 0.019s
I20250115 02:53:04.228657 8669 raft_consensus.cc:357] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.229384 8680 raft_consensus.cc:513] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.230310 8673 tablet_bootstrap.cc:492] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94: Bootstrap starting.
I20250115 02:53:04.232101 8467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2e395c8a0149ae9c88f74272f7b8df" candidate_uuid: "6e224d15a8964126b95c016aa46efb28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94"
I20250115 02:53:04.231417 8677 leader_election.cc:290] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [CANDIDATE]: Term 1 election: Requested vote from peers c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713), 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65:36051)
W20250115 02:53:04.233249 8588 leader_election.cc:343] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65:36051): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:04.235054 8394 catalog_manager.cc:1260] Loaded cluster ID: e2d47ad2480e4ddaa1f0b56f7191cea2
I20250115 02:53:04.235357 8394 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: loading cluster ID for follower catalog manager: success
I20250115 02:53:04.229295 8545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2e395c8a0149ae9c88f74272f7b8df" candidate_uuid: "6e224d15a8964126b95c016aa46efb28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04b74f0e5444b2f94c83d82c89c8426"
I20250115 02:53:04.237267 8545 raft_consensus.cc:3054] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:04.239276 8673 tablet_bootstrap.cc:654] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:04.242684 8394 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: acquiring CA information for follower catalog manager: success
I20250115 02:53:04.249275 8394 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:04.229554 8669 raft_consensus.cc:383] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:04.249820 8669 raft_consensus.cc:738] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e224d15a8964126b95c016aa46efb28, State: Initialized, Role: FOLLOWER
I20250115 02:53:04.250931 8673 tablet_bootstrap.cc:492] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94: No bootstrap required, opened a new log
I20250115 02:53:04.250567 8669 consensus_queue.cc:260] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.253054 8669 ts_tablet_manager.cc:1428] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28: Time spent starting tablet: real 0.027s user 0.009s sys 0.000s
I20250115 02:53:04.253495 8673 ts_tablet_manager.cc:1397] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.006s
I20250115 02:53:04.254094 8669 tablet_bootstrap.cc:492] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28: Bootstrap starting.
I20250115 02:53:04.257323 8673 raft_consensus.cc:357] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.258219 8673 raft_consensus.cc:383] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:04.258684 8673 raft_consensus.cc:738] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2b237918cd4b8c9cb8ff9cf1356f94, State: Initialized, Role: FOLLOWER
I20250115 02:53:04.259294 8545 raft_consensus.cc:2463] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e224d15a8964126b95c016aa46efb28 in term 1.
I20250115 02:53:04.259464 8673 consensus_queue.cc:260] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.260829 8669 tablet_bootstrap.cc:654] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:04.261924 8586 leader_election.cc:304] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e224d15a8964126b95c016aa46efb28, c04b74f0e5444b2f94c83d82c89c8426; no voters: 3a2b237918cd4b8c9cb8ff9cf1356f94
I20250115 02:53:04.262046 8673 ts_tablet_manager.cc:1428] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250115 02:53:04.262635 8677 raft_consensus.cc:2798] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:04.263154 8677 raft_consensus.cc:695] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 1 LEADER]: Becoming Leader. State: Replica: 6e224d15a8964126b95c016aa46efb28, State: Running, Role: LEADER
I20250115 02:53:04.263619 8673 tablet_bootstrap.cc:492] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94: Bootstrap starting.
I20250115 02:53:04.263990 8677 consensus_queue.cc:237] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.268137 8407 catalog_manager.cc:1260] Loaded cluster ID: e2d47ad2480e4ddaa1f0b56f7191cea2
I20250115 02:53:04.268435 8407 catalog_manager.cc:1553] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: loading cluster ID for follower catalog manager: success
I20250115 02:53:04.274482 8673 tablet_bootstrap.cc:654] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:04.276175 8407 catalog_manager.cc:1575] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: acquiring CA information for follower catalog manager: success
I20250115 02:53:04.284755 8680 leader_election.cc:290] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713), 6e224d15a8964126b95c016aa46efb28 (127.2.36.67:38517)
I20250115 02:53:04.285629 8673 tablet_bootstrap.cc:492] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94: No bootstrap required, opened a new log
I20250115 02:53:04.286325 8673 ts_tablet_manager.cc:1397] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.003s
I20250115 02:53:04.287117 8407 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:04.292258 8673 raft_consensus.cc:357] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.294541 8673 raft_consensus.cc:383] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:04.294896 8673 raft_consensus.cc:738] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2b237918cd4b8c9cb8ff9cf1356f94, State: Initialized, Role: FOLLOWER
I20250115 02:53:04.295786 8669 tablet_bootstrap.cc:492] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28: No bootstrap required, opened a new log
I20250115 02:53:04.296397 8669 ts_tablet_manager.cc:1397] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.004s
I20250115 02:53:04.295742 8673 consensus_queue.cc:260] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.309644 8669 raft_consensus.cc:357] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.310456 8669 raft_consensus.cc:383] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:04.310781 8669 raft_consensus.cc:738] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e224d15a8964126b95c016aa46efb28, State: Initialized, Role: FOLLOWER
I20250115 02:53:04.311760 8673 ts_tablet_manager.cc:1428] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20250115 02:53:04.311522 8669 consensus_queue.cc:260] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.312804 8200 catalog_manager.cc:5526] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e224d15a8964126b95c016aa46efb28 (127.2.36.67). New cstate: current_term: 1 leader_uuid: "6e224d15a8964126b95c016aa46efb28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } health_report { overall_health: HEALTHY } } }
I20250115 02:53:04.323355 8669 ts_tablet_manager.cc:1428] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250115 02:53:04.324038 8545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c0a914505c44b59cc22b05ed63518a" candidate_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04b74f0e5444b2f94c83d82c89c8426" is_pre_election: true
I20250115 02:53:04.324712 8545 raft_consensus.cc:2463] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2b237918cd4b8c9cb8ff9cf1356f94 in term 0.
I20250115 02:53:04.325750 8420 leader_election.cc:304] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2b237918cd4b8c9cb8ff9cf1356f94, c04b74f0e5444b2f94c83d82c89c8426; no voters:
I20250115 02:53:04.326581 8680 raft_consensus.cc:2798] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:04.326989 8680 raft_consensus.cc:491] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:04.327327 8680 raft_consensus.cc:3054] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:04.340706 8680 raft_consensus.cc:513] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.344264 8545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c0a914505c44b59cc22b05ed63518a" candidate_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04b74f0e5444b2f94c83d82c89c8426"
I20250115 02:53:04.344908 8545 raft_consensus.cc:3054] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:04.345201 8680 leader_election.cc:290] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [CANDIDATE]: Term 1 election: Requested vote from peers c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713), 6e224d15a8964126b95c016aa46efb28 (127.2.36.67:38517)
I20250115 02:53:04.351502 8622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c0a914505c44b59cc22b05ed63518a" candidate_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e224d15a8964126b95c016aa46efb28"
I20250115 02:53:04.352207 8622 raft_consensus.cc:3054] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:04.354126 8545 raft_consensus.cc:2463] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2b237918cd4b8c9cb8ff9cf1356f94 in term 1.
I20250115 02:53:04.351567 8623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c0a914505c44b59cc22b05ed63518a" candidate_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e224d15a8964126b95c016aa46efb28" is_pre_election: true
I20250115 02:53:04.355180 8420 leader_election.cc:304] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2b237918cd4b8c9cb8ff9cf1356f94, c04b74f0e5444b2f94c83d82c89c8426; no voters:
I20250115 02:53:04.357326 8680 raft_consensus.cc:2798] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:04.357803 8680 raft_consensus.cc:695] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 LEADER]: Becoming Leader. State: Replica: 3a2b237918cd4b8c9cb8ff9cf1356f94, State: Running, Role: LEADER
I20250115 02:53:04.358650 8680 consensus_queue.cc:237] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.369165 8622 raft_consensus.cc:2463] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2b237918cd4b8c9cb8ff9cf1356f94 in term 1.
I20250115 02:53:04.381548 8199 catalog_manager.cc:5526] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:04.427734 8680 raft_consensus.cc:491] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:04.428645 8680 raft_consensus.cc:513] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.430284 8680 leader_election.cc:290] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713), 6e224d15a8964126b95c016aa46efb28 (127.2.36.67:38517)
I20250115 02:53:04.431301 8545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb519c79297400785d3ce4bb2fc232e" candidate_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04b74f0e5444b2f94c83d82c89c8426" is_pre_election: true
I20250115 02:53:04.432021 8545 raft_consensus.cc:2463] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2b237918cd4b8c9cb8ff9cf1356f94 in term 0.
I20250115 02:53:04.433327 8622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb519c79297400785d3ce4bb2fc232e" candidate_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e224d15a8964126b95c016aa46efb28" is_pre_election: true
I20250115 02:53:04.433439 8420 leader_election.cc:304] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2b237918cd4b8c9cb8ff9cf1356f94, c04b74f0e5444b2f94c83d82c89c8426; no voters:
I20250115 02:53:04.434053 8622 raft_consensus.cc:2463] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2b237918cd4b8c9cb8ff9cf1356f94 in term 0.
I20250115 02:53:04.434391 8680 raft_consensus.cc:2798] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:04.434839 8680 raft_consensus.cc:491] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:04.435174 8680 raft_consensus.cc:3054] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:04.444491 8680 raft_consensus.cc:513] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.447146 8545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb519c79297400785d3ce4bb2fc232e" candidate_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04b74f0e5444b2f94c83d82c89c8426"
I20250115 02:53:04.447576 8680 leader_election.cc:290] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [CANDIDATE]: Term 1 election: Requested vote from peers c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713), 6e224d15a8964126b95c016aa46efb28 (127.2.36.67:38517)
I20250115 02:53:04.448294 8622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb519c79297400785d3ce4bb2fc232e" candidate_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e224d15a8964126b95c016aa46efb28"
I20250115 02:53:04.448885 8622 raft_consensus.cc:3054] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:04.447780 8545 raft_consensus.cc:3054] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:04.463867 8545 raft_consensus.cc:2463] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2b237918cd4b8c9cb8ff9cf1356f94 in term 1.
I20250115 02:53:04.464954 8420 leader_election.cc:304] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2b237918cd4b8c9cb8ff9cf1356f94, c04b74f0e5444b2f94c83d82c89c8426; no voters:
I20250115 02:53:04.466532 8622 raft_consensus.cc:2463] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2b237918cd4b8c9cb8ff9cf1356f94 in term 1.
I20250115 02:53:04.466935 8680 raft_consensus.cc:2798] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:04.467458 8680 raft_consensus.cc:695] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 LEADER]: Becoming Leader. State: Replica: 3a2b237918cd4b8c9cb8ff9cf1356f94, State: Running, Role: LEADER
I20250115 02:53:04.468255 8680 consensus_queue.cc:237] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.478433 8199 catalog_manager.cc:5526] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:04.594739 2193 master.cc:537] Master@127.2.36.126:43155 shutting down...
I20250115 02:53:04.639655 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:04.640708 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:04.641261 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: stopping tablet replica
I20250115 02:53:04.653131 8680 raft_consensus.cc:491] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:04.653713 8680 raft_consensus.cc:513] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 } } peers { permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 } } peers { permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 } }
I20250115 02:53:04.655643 8680 leader_election.cc:290] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713), 6e224d15a8964126b95c016aa46efb28 (127.2.36.67:38517)
I20250115 02:53:04.657987 8545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2e395c8a0149ae9c88f74272f7b8df" candidate_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04b74f0e5444b2f94c83d82c89c8426" is_pre_election: true
I20250115 02:53:04.658857 8545 raft_consensus.cc:2388] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a2b237918cd4b8c9cb8ff9cf1356f94 in current term 1: Already voted for candidate 6e224d15a8964126b95c016aa46efb28 in this term.
I20250115 02:53:04.659828 8622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2e395c8a0149ae9c88f74272f7b8df" candidate_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e224d15a8964126b95c016aa46efb28" is_pre_election: true
I20250115 02:53:04.663942 2193 master.cc:559] Master@127.2.36.126:43155 shutdown complete.
I20250115 02:53:04.663887 8420 leader_election.cc:304] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a2b237918cd4b8c9cb8ff9cf1356f94; no voters: 6e224d15a8964126b95c016aa46efb28, c04b74f0e5444b2f94c83d82c89c8426
I20250115 02:53:04.669175 8680 raft_consensus.cc:3054] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:04.680910 8677 consensus_queue.cc:1035] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:04.682904 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:04.686102 8680 raft_consensus.cc:2743] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 02:53:04.697842 8697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:04.706442 8698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:04.715237 8677 consensus_queue.cc:1035] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 36051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250115 02:53:04.728094 8702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:04.754639 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:04.756572 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:04.756886 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:04.757097 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909584757077 us; error 0 us; skew 500 ppm
I20250115 02:53:04.757848 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:04.768793 2193 webserver.cc:458] Webserver started at http://127.2.36.126:39727/ using document root <none> and password file <none>
I20250115 02:53:04.769330 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:04.769511 2193 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250115 02:53:04.776021 2193 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.002s
I20250115 02:53:04.776021 8680 consensus_queue.cc:1035] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 02:53:04.785682 8710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:04.786825 2193 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20250115 02:53:04.787191 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "412cd59ec6454237b54608b8639fd2cb"
format_stamp: "Formatted at 2025-01-15 02:53:02 on dist-test-slave-1lv2"
I20250115 02:53:04.787541 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:04.802357 8681 consensus_queue.cc:1035] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:04.819361 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:04.821169 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:04.912904 8680 consensus_queue.cc:1035] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e224d15a8964126b95c016aa46efb28" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:04.939771 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:43155
I20250115 02:53:04.939877 8763 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:43155 every 8 connection(s)
I20250115 02:53:04.956890 8680 consensus_queue.cc:1035] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 46713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:04.983193 8572 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43155
I20250115 02:53:04.988134 8495 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43155
I20250115 02:53:05.011397 8650 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43155
I20250115 02:53:05.057088 8768 sys_catalog.cc:263] Verifying existing consensus state
I20250115 02:53:05.059166 2193 master.cc:537] Master@127.2.36.125:34261 shutting down...
I20250115 02:53:05.068215 8768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: Bootstrap starting.
I20250115 02:53:05.070010 8572 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43155
I20250115 02:53:05.084874 8650 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43155
I20250115 02:53:05.122583 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:05.123266 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:05.123675 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: stopping tablet replica
W20250115 02:53:05.141506 8586 connection.cc:537] client connection to 127.2.36.125:34261 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250115 02:53:05.149066 8768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=7 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250115 02:53:05.150120 8768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: Bootstrap complete.
I20250115 02:53:05.150733 8495 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43155
I20250115 02:53:05.153250 8768 raft_consensus.cc:357] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:05.154089 8768 raft_consensus.cc:738] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412cd59ec6454237b54608b8639fd2cb, State: Initialized, Role: FOLLOWER
I20250115 02:53:05.154852 8768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:05.160797 2193 master.cc:559] Master@127.2.36.125:34261 shutdown complete.
I20250115 02:53:05.173563 8772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:05.176623 8768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:05.177181 8772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:05.184286 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:05.201536 8774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:05.206804 8776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:05.213323 8780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:05.226997 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:05.229049 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:05.229306 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:05.229499 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909585229482 us; error 0 us; skew 500 ppm
I20250115 02:53:05.230598 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:05.247316 8727 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" instance_seqno: 1736909583387681) as {username='slave'} at 127.0.0.1:56350; Asking this server to re-register.
I20250115 02:53:05.249577 8495 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:05.247259 8728 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6e224d15a8964126b95c016aa46efb28" instance_seqno: 1736909583682550) as {username='slave'} at 127.0.0.1:56338; Asking this server to re-register.
I20250115 02:53:05.252561 8650 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:05.366724 8726 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" instance_seqno: 1736909583535969) as {username='slave'} at 127.0.0.1:56366; Asking this server to re-register.
I20250115 02:53:05.371290 8788 catalog_manager.cc:1260] Loaded cluster ID: e2d47ad2480e4ddaa1f0b56f7191cea2
I20250115 02:53:05.371814 8788 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: loading cluster ID for follower catalog manager: success
W20250115 02:53:05.424072 8728 subprocess.cc:277] Child process 8791 (/tmp/dist-test-taskB0ERDW/build/tsan/bin/testdata/first_argument.sh /foo 127.2.36.65) was orphaned. Sending signal 9...
W20250115 02:53:05.430608 8728 subprocess.cc:282] Failed to KillAndWait() with signal 9: Runtime error: Failed to send signal 9 to /tmp/dist-test-taskB0ERDW/build/tsan/bin/testdata/first_argument.sh (pid 8791): Unable to kill: No such process (error 3)
E20250115 02:53:05.432825 8728 ts_manager.cc:169] Unable to assign location to tablet server 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65:36051): Runtime error: failed to run location mapping command: Runtime error: Unable to wait() for /tmp/dist-test-taskB0ERDW/build/tsan/bin/testdata/first_argument.sh: Unable to wait on child: No child processes (error 10)
W20250115 02:53:05.433919 8728 master_service.cc:423] Unable to register tserver ({username='slave'} at 127.0.0.1:56350): Runtime error: failed to run location mapping command: Runtime error: Unable to wait() for /tmp/dist-test-taskB0ERDW/build/tsan/bin/testdata/first_argument.sh: Unable to wait on child: No child processes (error 10)
I20250115 02:53:05.435593 2193 webserver.cc:458] Webserver started at http://127.2.36.125:38501/ using document root <none> and password file <none>
I20250115 02:53:05.450086 8495 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:05.456053 8572 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:05.458333 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:05.458942 2193 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250115 02:53:05.570310 8727 ts_manager.cc:194] Registered new tserver with Master: 6e224d15a8964126b95c016aa46efb28 (127.2.36.67:38517)
I20250115 02:53:05.602238 8788 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: acquiring CA information for follower catalog manager: success
I20250115 02:53:05.751124 8788 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:05.788980 8728 ts_manager.cc:194] Registered new tserver with Master: c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713)
I20250115 02:53:05.872983 2193 fs_manager.cc:714] Time spent opening directory manager: real 0.410s user 0.095s sys 0.119s
I20250115 02:53:05.872983 8726 ts_manager.cc:194] Registered new tserver with Master: 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65:36051)
I20250115 02:53:05.883302 8796 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:05.886672 2193 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20250115 02:53:05.887053 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "e8b6772d16f44e0f9ad5d9936490d280"
format_stamp: "Formatted at 2025-01-15 02:53:02 on dist-test-slave-1lv2"
I20250115 02:53:05.887415 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:05.919669 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:05.921316 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:06.018787 8837 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 412cd59ec6454237b54608b8639fd2cb)
I20250115 02:53:06.019568 8837 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:06.038285 8837 leader_election.cc:290] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 412cd59ec6454237b54608b8639fd2cb (127.2.36.126:43155), e8b6772d16f44e0f9ad5d9936490d280 (127.2.36.125:34261)
I20250115 02:53:06.059595 8738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6352b3d955204093b104d1efc24ebc88" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "412cd59ec6454237b54608b8639fd2cb" is_pre_election: true
I20250115 02:53:06.060590 8738 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6352b3d955204093b104d1efc24ebc88 in term 1.
I20250115 02:53:06.061398 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:34261
I20250115 02:53:06.061600 8852 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:34261 every 8 connection(s)
I20250115 02:53:06.062050 8314 leader_election.cc:304] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412cd59ec6454237b54608b8639fd2cb, 6352b3d955204093b104d1efc24ebc88; no voters:
I20250115 02:53:06.062994 8837 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:53:06.063365 8837 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 412cd59ec6454237b54608b8639fd2cb)
I20250115 02:53:06.063711 8837 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:06.085165 8837 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:06.087885 8837 leader_election.cc:290] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [CANDIDATE]: Term 2 election: Requested vote from peers 412cd59ec6454237b54608b8639fd2cb (127.2.36.126:43155), e8b6772d16f44e0f9ad5d9936490d280 (127.2.36.125:34261)
I20250115 02:53:06.090152 8738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6352b3d955204093b104d1efc24ebc88" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "412cd59ec6454237b54608b8639fd2cb"
I20250115 02:53:06.090867 8738 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:06.119431 8738 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6352b3d955204093b104d1efc24ebc88 in term 2.
I20250115 02:53:06.125973 8314 leader_election.cc:304] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412cd59ec6454237b54608b8639fd2cb, 6352b3d955204093b104d1efc24ebc88; no voters:
I20250115 02:53:06.128757 8837 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 2 FOLLOWER]: Leader election won for term 2
I20250115 02:53:06.136303 8570 heartbeater.cc:346] Connected to a master server at 127.2.36.125:34261
I20250115 02:53:06.142577 8492 heartbeater.cc:346] Connected to a master server at 127.2.36.125:34261
I20250115 02:53:06.149834 8855 sys_catalog.cc:263] Verifying existing consensus state
I20250115 02:53:06.151021 8648 heartbeater.cc:346] Connected to a master server at 127.2.36.125:34261
I20250115 02:53:06.161885 2193 master.cc:537] Master@127.2.36.124:39147 shutting down...
I20250115 02:53:06.174409 8855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: Bootstrap starting.
I20250115 02:53:06.240360 8837 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 2 LEADER]: Becoming Leader. State: Replica: 6352b3d955204093b104d1efc24ebc88, State: Running, Role: LEADER
I20250115 02:53:06.241559 8837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:06.256538 8859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6352b3d955204093b104d1efc24ebc88. Latest consensus state: current_term: 2 leader_uuid: "6352b3d955204093b104d1efc24ebc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:06.257428 8859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:06.274861 8862 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:06.281973 8862 catalog_manager.cc:670] Loaded metadata for table ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f]
I20250115 02:53:06.305573 8862 tablet_loader.cc:96] loaded metadata for tablet 1e2e395c8a0149ae9c88f74272f7b8df (table ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f])
I20250115 02:53:06.307221 8862 tablet_loader.cc:96] loaded metadata for tablet 7eb519c79297400785d3ce4bb2fc232e (table ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f])
I20250115 02:53:06.308625 8862 tablet_loader.cc:96] loaded metadata for tablet f2c0a914505c44b59cc22b05ed63518a (table ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f])
I20250115 02:53:06.310217 8862 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:06.317734 8862 catalog_manager.cc:1260] Loaded cluster ID: e2d47ad2480e4ddaa1f0b56f7191cea2
I20250115 02:53:06.318048 8862 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:06.323743 8862 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:06.328696 8862 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: Loaded TSK: 0
I20250115 02:53:06.329806 8862 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:06.337594 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 2 LEADER]: Raft consensus shutting down.
I20250115 02:53:06.338589 2193 pending_rounds.cc:62] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: Trying to abort 1 pending ops.
I20250115 02:53:06.338816 8855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=7 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250115 02:53:06.338943 2193 pending_rounds.cc:69] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: Aborting op as it isn't in flight: id { term: 2 index: 10 } timestamp: 7114381665257025536 op_type: NO_OP noop_request { }
I20250115 02:53:06.339594 2193 raft_consensus.cc:2883] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 02:53:06.339991 8855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: Bootstrap complete.
I20250115 02:53:06.339994 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:06.341393 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: stopping tablet replica
I20250115 02:53:06.352212 8855 raft_consensus.cc:357] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:06.353029 8855 raft_consensus.cc:738] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b6772d16f44e0f9ad5d9936490d280, State: Initialized, Role: FOLLOWER
W20250115 02:53:06.355078 8313 leader_election.cc:336] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e8b6772d16f44e0f9ad5d9936490d280 (127.2.36.125:34261): Service unavailable: reactor is shutting down (error 108)
W20250115 02:53:06.355540 8849 leader_election.cc:336] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e8b6772d16f44e0f9ad5d9936490d280 (127.2.36.125:34261): Service unavailable: reactor is shutting down (error 108)
I20250115 02:53:06.353695 8855 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:06.365658 8855 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:06.366719 8823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6352b3d955204093b104d1efc24ebc88" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e8b6772d16f44e0f9ad5d9936490d280" is_pre_election: true
I20250115 02:53:06.367477 8823 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6352b3d955204093b104d1efc24ebc88 in term 1.
W20250115 02:53:06.368275 8798 connection.cc:441] server connection from 127.0.0.1:34746 torn down before Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34746 (request call id 0) could send its response
I20250115 02:53:06.369036 8822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6352b3d955204093b104d1efc24ebc88" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e8b6772d16f44e0f9ad5d9936490d280"
I20250115 02:53:06.369685 8822 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:06.384526 2193 master.cc:559] Master@127.2.36.124:39147 shutdown complete.
I20250115 02:53:06.387342 8822 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6352b3d955204093b104d1efc24ebc88 in term 2.
W20250115 02:53:06.388197 8798 connection.cc:441] server connection from 127.0.0.1:34746 torn down before Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34746 (request call id 1) could send its response
I20250115 02:53:06.393230 8863 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:06.394392 8863 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:06.400496 8570 heartbeater.cc:346] Connected to a master server at 127.2.36.125:34261
I20250115 02:53:06.426618 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:06.434434 8874 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250115 02:53:06.438642 8875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:06.441429 8876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:06.438185 8813 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" instance_seqno: 1736909583535969) as {username='slave'} at 127.0.0.1:34760; Asking this server to re-register.
I20250115 02:53:06.443133 8570 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:06.446012 8812 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" instance_seqno: 1736909583387681) as {username='slave'} at 127.0.0.1:34768; Asking this server to re-register.
I20250115 02:53:06.448657 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:06.448962 8492 heartbeater.cc:463] Registering TS with master...
W20250115 02:53:06.449851 8878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:06.451077 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:06.451383 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:06.451627 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909586451607 us; error 0 us; skew 500 ppm
I20250115 02:53:06.452265 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:06.604082 8812 ts_manager.cc:194] Registered new tserver with Master: 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65:36051)
I20250115 02:53:06.712457 8811 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6e224d15a8964126b95c016aa46efb28" instance_seqno: 1736909583682550) as {username='slave'} at 127.0.0.1:34778; Asking this server to re-register.
I20250115 02:53:06.720134 8648 heartbeater.cc:463] Registering TS with master...
W20250115 02:53:06.818025 8813 subprocess.cc:277] Child process 8882 (/tmp/dist-test-taskB0ERDW/build/tsan/bin/testdata/first_argument.sh /foo 127.2.36.66) was orphaned. Sending signal 9...
W20250115 02:53:06.832792 8813 subprocess.cc:282] Failed to KillAndWait() with signal 9: Runtime error: Failed to send signal 9 to /tmp/dist-test-taskB0ERDW/build/tsan/bin/testdata/first_argument.sh (pid 8882): Unable to kill: No such process (error 3)
W20250115 02:53:06.834789 8813 master_service.cc:423] Unable to register tserver ({username='slave'} at 127.0.0.1:34760): Runtime error: failed to run location mapping command: Runtime error: Unable to wait() for /tmp/dist-test-taskB0ERDW/build/tsan/bin/testdata/first_argument.sh: Unable to wait on child: No child processes (error 10)
I20250115 02:53:06.843752 8570 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:06.868180 2193 webserver.cc:458] Webserver started at http://127.2.36.124:40371/ using document root <none> and password file <none>
I20250115 02:53:07.085026 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:07.085880 2193 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250115 02:53:07.137965 8812 ts_manager.cc:194] Registered new tserver with Master: 6e224d15a8964126b95c016aa46efb28 (127.2.36.67:38517)
I20250115 02:53:07.153187 8811 ts_manager.cc:194] Registered new tserver with Master: c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713)
I20250115 02:53:07.157538 2193 fs_manager.cc:714] Time spent opening directory manager: real 0.018s user 0.004s sys 0.004s
I20250115 02:53:07.172710 8890 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:07.173615 2193 fs_manager.cc:730] Time spent opening block manager: real 0.013s user 0.005s sys 0.003s
I20250115 02:53:07.173959 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "6352b3d955204093b104d1efc24ebc88"
format_stamp: "Formatted at 2025-01-15 02:53:03 on dist-test-slave-1lv2"
I20250115 02:53:07.174290 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:07.213706 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:07.215312 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:07.336095 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:39147
I20250115 02:53:07.336236 8941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:39147 every 8 connection(s)
I20250115 02:53:07.353119 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:07.356433 8942 sys_catalog.cc:263] Verifying existing consensus state
I20250115 02:53:07.365762 8942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: Bootstrap starting.
W20250115 02:53:07.371593 8946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:07.381934 8947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:07.392959 8571 heartbeater.cc:346] Connected to a master server at 127.2.36.124:39147
I20250115 02:53:07.395385 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:07.399286 8949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:07.400609 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:07.400939 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:07.401167 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909587401150 us; error 0 us; skew 500 ppm
I20250115 02:53:07.402112 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:07.412719 8649 heartbeater.cc:346] Connected to a master server at 127.2.36.124:39147
I20250115 02:53:07.414880 2193 webserver.cc:458] Webserver started at http://127.2.36.68:37131/ using document root <none> and password file <none>
I20250115 02:53:07.415757 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:07.416019 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:07.416407 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:07.418345 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-3-root/instance:
uuid: "de159bf4a3ef4c4ab849980ebe2e66e3"
format_stamp: "Formatted at 2025-01-15 02:53:07 on dist-test-slave-1lv2"
I20250115 02:53:07.428342 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20250115 02:53:07.440925 8874 catalog_manager.cc:1260] Loaded cluster ID: e2d47ad2480e4ddaa1f0b56f7191cea2
I20250115 02:53:07.441546 8874 catalog_manager.cc:1553] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: loading cluster ID for follower catalog manager: success
I20250115 02:53:07.442704 8493 heartbeater.cc:346] Connected to a master server at 127.2.36.124:39147
I20250115 02:53:07.447031 8942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=9 ignored=0} inserts{seen=7 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250115 02:53:07.448194 8942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: Bootstrap complete.
I20250115 02:53:07.450294 8874 catalog_manager.cc:1575] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: acquiring CA information for follower catalog manager: success
I20250115 02:53:07.451689 8942 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:07.452795 8942 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6352b3d955204093b104d1efc24ebc88, State: Initialized, Role: FOLLOWER
I20250115 02:53:07.453140 8955 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:07.454243 2193 fs_manager.cc:730] Time spent opening block manager: real 0.020s user 0.002s sys 0.001s
I20250115 02:53:07.453737 8942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:07.454716 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-3-root
uuid: "de159bf4a3ef4c4ab849980ebe2e66e3"
format_stamp: "Formatted at 2025-01-15 02:53:07 on dist-test-slave-1lv2"
I20250115 02:53:07.455214 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestClusterWithLocation.1736909525896969-2193-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:07.456656 8874 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:07.458320 8956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:07.459527 8956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:07.460124 8942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:07.506676 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:07.508646 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:07.510998 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:07.514259 8971 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250115 02:53:07.516407 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:07.516798 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:07.517174 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:07.517416 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:07.526475 8907 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3a2b237918cd4b8c9cb8ff9cf1356f94" instance_seqno: 1736909583387681) as {username='slave'} at 127.0.0.1:58316; Asking this server to re-register.
I20250115 02:53:07.529739 8493 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:07.530860 8907 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6e224d15a8964126b95c016aa46efb28" instance_seqno: 1736909583682550) as {username='slave'} at 127.0.0.1:58306; Asking this server to re-register.
I20250115 02:53:07.532091 8649 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:07.654325 8905 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c04b74f0e5444b2f94c83d82c89c8426" instance_seqno: 1736909583535969) as {username='slave'} at 127.0.0.1:58304; Asking this server to re-register.
I20250115 02:53:07.719781 8571 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:07.898730 8978 raft_consensus.cc:491] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:07.929067 8978 raft_consensus.cc:513] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:07.910709 8907 ts_manager.cc:194] Registered new tserver with Master: 3a2b237918cd4b8c9cb8ff9cf1356f94 (127.2.36.65:36051)
I20250115 02:53:07.954114 8978 leader_election.cc:290] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e8b6772d16f44e0f9ad5d9936490d280 (127.2.36.125:34261), 6352b3d955204093b104d1efc24ebc88 (127.2.36.124:39147)
I20250115 02:53:07.956435 8906 ts_manager.cc:194] Registered new tserver with Master: 6e224d15a8964126b95c016aa46efb28 (127.2.36.67:38517)
I20250115 02:53:07.959008 8905 ts_manager.cc:194] Registered new tserver with Master: c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713)
I20250115 02:53:08.058580 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "412cd59ec6454237b54608b8639fd2cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6352b3d955204093b104d1efc24ebc88" is_pre_election: true
I20250115 02:53:08.059913 8917 raft_consensus.cc:2405] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 412cd59ec6454237b54608b8639fd2cb for term 3 because replica has last-logged OpId of term: 2 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 9.
I20250115 02:53:08.060576 8822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "412cd59ec6454237b54608b8639fd2cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e8b6772d16f44e0f9ad5d9936490d280" is_pre_election: true
I20250115 02:53:08.061363 8822 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412cd59ec6454237b54608b8639fd2cb in term 2.
I20250115 02:53:08.062768 8711 leader_election.cc:304] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 412cd59ec6454237b54608b8639fd2cb, e8b6772d16f44e0f9ad5d9936490d280; no voters: 6352b3d955204093b104d1efc24ebc88
I20250115 02:53:08.064488 8978 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250115 02:53:08.064889 8978 raft_consensus.cc:491] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:08.065160 8978 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:53:08.111871 8978 raft_consensus.cc:513] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:08.117231 8822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "412cd59ec6454237b54608b8639fd2cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e8b6772d16f44e0f9ad5d9936490d280"
I20250115 02:53:08.117897 8822 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:53:08.121162 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "412cd59ec6454237b54608b8639fd2cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6352b3d955204093b104d1efc24ebc88"
I20250115 02:53:08.121882 8917 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:53:08.128937 8822 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412cd59ec6454237b54608b8639fd2cb in term 3.
I20250115 02:53:08.131300 8711 leader_election.cc:304] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412cd59ec6454237b54608b8639fd2cb, e8b6772d16f44e0f9ad5d9936490d280; no voters:
I20250115 02:53:08.132237 8978 leader_election.cc:290] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [CANDIDATE]: Term 3 election: Requested vote from peers e8b6772d16f44e0f9ad5d9936490d280 (127.2.36.125:34261), 6352b3d955204093b104d1efc24ebc88 (127.2.36.124:39147)
I20250115 02:53:08.133937 8978 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 3 FOLLOWER]: Leader election won for term 3
I20250115 02:53:08.134449 8978 raft_consensus.cc:695] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 3 LEADER]: Becoming Leader. State: Replica: 412cd59ec6454237b54608b8639fd2cb, State: Running, Role: LEADER
I20250115 02:53:08.135407 8978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:08.151405 8917 raft_consensus.cc:2405] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 412cd59ec6454237b54608b8639fd2cb for term 3 because replica has last-logged OpId of term: 2 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 9.
I20250115 02:53:08.155807 8991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412cd59ec6454237b54608b8639fd2cb. Latest consensus state: current_term: 3 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:08.156661 8991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:08.183979 8995 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:08.191785 8995 catalog_manager.cc:670] Loaded metadata for table ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f]
I20250115 02:53:08.198017 8995 tablet_loader.cc:96] loaded metadata for tablet 1e2e395c8a0149ae9c88f74272f7b8df (table ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f])
I20250115 02:53:08.208146 8995 tablet_loader.cc:96] loaded metadata for tablet 7eb519c79297400785d3ce4bb2fc232e (table ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f])
I20250115 02:53:08.209692 8995 tablet_loader.cc:96] loaded metadata for tablet f2c0a914505c44b59cc22b05ed63518a (table ksck-test-table [id=7b57cd03259e414396f538b7b8719d4f])
I20250115 02:53:08.211370 8995 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:08.225203 8995 catalog_manager.cc:1260] Loaded cluster ID: e2d47ad2480e4ddaa1f0b56f7191cea2
I20250115 02:53:08.225607 8995 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:08.241379 8995 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:08.253971 8995 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb: Loaded TSK: 0
I20250115 02:53:08.255249 8995 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:08.285921 8572 heartbeater.cc:501] Master 127.2.36.126:43155 was elected leader, sending a full tablet report...
I20250115 02:53:08.292212 8650 heartbeater.cc:501] Master 127.2.36.126:43155 was elected leader, sending a full tablet report...
I20250115 02:53:08.296780 8495 heartbeater.cc:501] Master 127.2.36.126:43155 was elected leader, sending a full tablet report...
I20250115 02:53:08.527689 8971 catalog_manager.cc:1260] Loaded cluster ID: e2d47ad2480e4ddaa1f0b56f7191cea2
I20250115 02:53:08.529100 8971 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: loading cluster ID for follower catalog manager: success
I20250115 02:53:08.550186 8971 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: acquiring CA information for follower catalog manager: success
I20250115 02:53:08.558573 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.68:41629
I20250115 02:53:08.559285 9040 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.68:41629 every 8 connection(s)
I20250115 02:53:08.571692 8971 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:08.647722 9041 heartbeater.cc:346] Connected to a master server at 127.2.36.125:34261
I20250115 02:53:08.648183 9041 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:08.649134 9041 heartbeater.cc:509] Master 127.2.36.125:34261 requested a full tablet report, sending...
I20250115 02:53:08.772595 9042 heartbeater.cc:346] Connected to a master server at 127.2.36.124:39147
I20250115 02:53:08.808140 9042 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:08.895941 9042 heartbeater.cc:509] Master 127.2.36.124:39147 requested a full tablet report, sending...
I20250115 02:53:08.908461 8917 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 3 FOLLOWER]: Refusing update from remote peer 412cd59ec6454237b54608b8639fd2cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 10. (term mismatch)
I20250115 02:53:08.916204 8917 pending_rounds.cc:77] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: Aborting all ops after (but not including) 9
I20250115 02:53:08.916741 8917 pending_rounds.cc:99] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88: Aborting uncommitted NO_OP operation due to leader change: 2.10
I20250115 02:53:08.912873 8822 raft_consensus.cc:1270] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 3 FOLLOWER]: Refusing update from remote peer 412cd59ec6454237b54608b8639fd2cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20250115 02:53:08.917099 8917 raft_consensus.cc:2883] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250115 02:53:08.919893 9053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20250115 02:53:09.008968 8420 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.36.66:46713, user_credentials={real_user=slave}} blocked reactor thread for 99369.1us
I20250115 02:53:09.031793 9053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250115 02:53:08.988667 8810 ts_manager.cc:194] Registered new tserver with Master: de159bf4a3ef4c4ab849980ebe2e66e3 (127.2.36.68:41629)
I20250115 02:53:09.145073 9045 heartbeater.cc:346] Connected to a master server at 127.2.36.126:43155
I20250115 02:53:09.145735 9045 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:09.147038 9045 heartbeater.cc:509] Master 127.2.36.126:43155 requested a full tablet report, sending...
I20250115 02:53:09.187505 8907 ts_manager.cc:194] Registered new tserver with Master: de159bf4a3ef4c4ab849980ebe2e66e3 (127.2.36.68:41629)
I20250115 02:53:09.398586 9057 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412cd59ec6454237b54608b8639fd2cb. Latest consensus state: current_term: 3 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:09.409404 9057 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:09.413692 9059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412cd59ec6454237b54608b8639fd2cb. Latest consensus state: current_term: 3 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:09.418270 9058 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 412cd59ec6454237b54608b8639fd2cb)
I20250115 02:53:09.418907 9058 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } }
I20250115 02:53:09.423597 9058 leader_election.cc:290] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 412cd59ec6454237b54608b8639fd2cb (127.2.36.126:43155), e8b6772d16f44e0f9ad5d9936490d280 (127.2.36.125:34261)
I20250115 02:53:09.414673 9059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:09.500641 8725 ts_manager.cc:194] Registered new tserver with Master: de159bf4a3ef4c4ab849980ebe2e66e3 (127.2.36.68:41629)
I20250115 02:53:09.527423 8725 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56382
I20250115 02:53:09.534650 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6352b3d955204093b104d1efc24ebc88" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "e8b6772d16f44e0f9ad5d9936490d280" is_pre_election: true
I20250115 02:53:09.542629 9045 heartbeater.cc:501] Master 127.2.36.126:43155 was elected leader, sending a full tablet report...
I20250115 02:53:09.547884 9059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:09.548961 9059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:09.567627 9057 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:09.568645 9057 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:09.572321 8737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6352b3d955204093b104d1efc24ebc88" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "412cd59ec6454237b54608b8639fd2cb" is_pre_election: true
I20250115 02:53:09.576229 8892 leader_election.cc:304] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6352b3d955204093b104d1efc24ebc88; no voters: 412cd59ec6454237b54608b8639fd2cb, e8b6772d16f44e0f9ad5d9936490d280
I20250115 02:53:09.577318 9059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250115 02:53:09.612257 9053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:09.615667 9053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:09.616698 9052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "412cd59ec6454237b54608b8639fd2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412cd59ec6454237b54608b8639fd2cb" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 43155 } } peers { permanent_uuid: "e8b6772d16f44e0f9ad5d9936490d280" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 34261 } } peers { permanent_uuid: "6352b3d955204093b104d1efc24ebc88" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 39147 } } }
I20250115 02:53:09.617631 9052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:10.556241 8454 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250115 02:53:10.605914 9006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:53:10.627787 8612 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:53:10.635900 8535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:53:11.216593 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:53:11.299912 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:11.300853 2193 tablet_replica.cc:331] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94: stopping tablet replica
I20250115 02:53:11.301680 2193 raft_consensus.cc:2238] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:11.302325 2193 raft_consensus.cc:2267] T 1e2e395c8a0149ae9c88f74272f7b8df P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:11.305353 2193 tablet_replica.cc:331] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94: stopping tablet replica
I20250115 02:53:11.306684 2193 raft_consensus.cc:2238] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:11.308279 2193 raft_consensus.cc:2267] T 7eb519c79297400785d3ce4bb2fc232e P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:11.311785 2193 tablet_replica.cc:331] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94: stopping tablet replica
I20250115 02:53:11.312604 2193 raft_consensus.cc:2238] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:11.313874 2193 raft_consensus.cc:2267] T f2c0a914505c44b59cc22b05ed63518a P 3a2b237918cd4b8c9cb8ff9cf1356f94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:11.332857 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:53:11.356603 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:53:11.408592 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:11.410686 2193 tablet_replica.cc:331] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426: stopping tablet replica
I20250115 02:53:11.411567 2193 raft_consensus.cc:2238] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:11.412107 2193 raft_consensus.cc:2267] T 1e2e395c8a0149ae9c88f74272f7b8df P c04b74f0e5444b2f94c83d82c89c8426 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:11.414359 2193 tablet_replica.cc:331] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426: stopping tablet replica
I20250115 02:53:11.414999 2193 raft_consensus.cc:2238] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:11.415485 2193 raft_consensus.cc:2267] T 7eb519c79297400785d3ce4bb2fc232e P c04b74f0e5444b2f94c83d82c89c8426 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:11.417894 2193 tablet_replica.cc:331] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426: stopping tablet replica
I20250115 02:53:11.418525 2193 raft_consensus.cc:2238] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:11.419001 2193 raft_consensus.cc:2267] T f2c0a914505c44b59cc22b05ed63518a P c04b74f0e5444b2f94c83d82c89c8426 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:11.444240 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:53:11.462864 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:53:11.505012 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:11.505869 2193 tablet_replica.cc:331] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28: stopping tablet replica
I20250115 02:53:11.506661 2193 raft_consensus.cc:2238] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:11.507195 2193 raft_consensus.cc:2267] T f2c0a914505c44b59cc22b05ed63518a P 6e224d15a8964126b95c016aa46efb28 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 02:53:11.508934 8586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.36.66:46713: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20250115 02:53:11.509990 2193 tablet_replica.cc:331] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28: stopping tablet replica
I20250115 02:53:11.510655 2193 raft_consensus.cc:2238] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:11.511212 2193 raft_consensus.cc:2267] T 7eb519c79297400785d3ce4bb2fc232e P 6e224d15a8964126b95c016aa46efb28 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 02:53:11.513252 8586 consensus_peers.cc:487] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 -> Peer c04b74f0e5444b2f94c83d82c89c8426 (127.2.36.66:46713): Couldn't send request to peer c04b74f0e5444b2f94c83d82c89c8426. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.66:46713: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:53:11.515331 2193 tablet_replica.cc:331] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28: stopping tablet replica
I20250115 02:53:11.516352 2193 raft_consensus.cc:2238] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:11.517550 2193 raft_consensus.cc:2267] T 1e2e395c8a0149ae9c88f74272f7b8df P 6e224d15a8964126b95c016aa46efb28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:11.532359 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:53:11.551463 2193 tablet_server.cc:178] TabletServer@127.2.36.68:0 shutting down...
I20250115 02:53:11.587314 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:11.604105 2193 tablet_server.cc:195] TabletServer@127.2.36.68:0 shutdown complete.
I20250115 02:53:11.614586 2193 master.cc:537] Master@127.2.36.126:43155 shutting down...
I20250115 02:53:11.636824 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 3 LEADER]: Raft consensus shutting down.
I20250115 02:53:11.637717 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 412cd59ec6454237b54608b8639fd2cb [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:11.638056 2193 tablet_replica.cc:331] stopping tablet replica
I20250115 02:53:11.659384 2193 master.cc:559] Master@127.2.36.126:43155 shutdown complete.
I20250115 02:53:11.676396 2193 master.cc:537] Master@127.2.36.125:34261 shutting down...
I20250115 02:53:11.696477 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:11.696957 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P e8b6772d16f44e0f9ad5d9936490d280 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:11.697229 2193 tablet_replica.cc:331] stopping tablet replica
I20250115 02:53:11.717479 2193 master.cc:559] Master@127.2.36.125:34261 shutdown complete.
I20250115 02:53:11.733562 2193 master.cc:537] Master@127.2.36.124:39147 shutting down...
I20250115 02:53:11.755442 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:11.755936 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 6352b3d955204093b104d1efc24ebc88 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:11.756232 2193 tablet_replica.cc:331] stopping tablet replica
I20250115 02:53:11.775856 2193 master.cc:559] Master@127.2.36.124:39147 shutdown complete.
[ OK ] RemoteKsckTest.TestClusterWithLocation (8923 ms)
[ RUN ] RemoteKsckTest.TestFilterOnNoTableCluster
I20250115 02:53:11.813668 2193 test_util.cc:274] Using random seed: -952492533
I20250115 02:53:11.814663 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:40435,127.2.36.125:43831,127.2.36.124:42199
I20250115 02:53:11.816037 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:11.822336 9111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:11.822904 9112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:11.824146 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:11.824760 9114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:11.825634 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:11.825843 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:11.825982 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909591825967 us; error 0 us; skew 500 ppm
I20250115 02:53:11.826467 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:11.833930 2193 webserver.cc:458] Webserver started at http://127.2.36.126:36625/ using document root <none> and password file <none>
I20250115 02:53:11.834434 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:11.834615 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:11.834900 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:11.836083 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "cb0280d4ae264c289de9481c1909c891"
format_stamp: "Formatted at 2025-01-15 02:53:11 on dist-test-slave-1lv2"
I20250115 02:53:11.840646 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:53:11.844029 9119 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:11.844761 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:11.845036 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "cb0280d4ae264c289de9481c1909c891"
format_stamp: "Formatted at 2025-01-15 02:53:11 on dist-test-slave-1lv2"
I20250115 02:53:11.845297 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:11.861610 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:11.862810 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:11.901203 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:40435
I20250115 02:53:11.901293 9170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:40435 every 8 connection(s)
I20250115 02:53:11.904985 9171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:11.905238 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:11.912714 9173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:11.915189 9171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:11.915875 9174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:11.919517 9176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:11.927086 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:11.928402 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:11.928674 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:11.928874 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909591928850 us; error 0 us; skew 500 ppm
I20250115 02:53:11.929646 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:11.933615 2193 webserver.cc:458] Webserver started at http://127.2.36.125:44917/ using document root <none> and password file <none>
I20250115 02:53:11.934274 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:11.934511 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:11.934870 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:11.936167 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "a612a825117d48bca9f5818edf5611ba"
format_stamp: "Formatted at 2025-01-15 02:53:11 on dist-test-slave-1lv2"
I20250115 02:53:11.941464 9171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:11.942242 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
W20250115 02:53:11.947533 9171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:43831: Network error: Client connection negotiation failed: client connection to 127.2.36.125:43831: connect: Connection refused (error 111)
I20250115 02:53:11.948427 9184 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:11.949381 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:53:11.949767 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "a612a825117d48bca9f5818edf5611ba"
format_stamp: "Formatted at 2025-01-15 02:53:11 on dist-test-slave-1lv2"
I20250115 02:53:11.950155 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:11.974328 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:11.975576 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:12.012003 9171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } attempt: 1
I20250115 02:53:12.015906 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:43831
I20250115 02:53:12.016055 9235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:43831 every 8 connection(s)
I20250115 02:53:12.021210 9237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:12.021132 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:12.028322 9171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:12.029717 9239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:12.031769 9237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:12.034894 9171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:42199: Network error: Client connection negotiation failed: client connection to 127.2.36.124:42199: connect: Connection refused (error 111)
W20250115 02:53:12.031780 9240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:12.037881 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:12.038981 9242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:12.039916 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:12.040123 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:12.040274 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909592040258 us; error 0 us; skew 500 ppm
I20250115 02:53:12.040807 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:12.043601 2193 webserver.cc:458] Webserver started at http://127.2.36.124:39637/ using document root <none> and password file <none>
I20250115 02:53:12.044123 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:12.044349 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:12.044661 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:12.045969 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "40adcf7887a74e9abc6cf7e462547caa"
format_stamp: "Formatted at 2025-01-15 02:53:12 on dist-test-slave-1lv2"
I20250115 02:53:12.046209 9237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:12.051054 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250115 02:53:12.055246 9248 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:12.056138 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250115 02:53:12.056417 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "40adcf7887a74e9abc6cf7e462547caa"
format_stamp: "Formatted at 2025-01-15 02:53:12 on dist-test-slave-1lv2"
I20250115 02:53:12.056591 9237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:12.056792 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250115 02:53:12.062135 9237 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:42199: Network error: Client connection negotiation failed: client connection to 127.2.36.124:42199: connect: Connection refused (error 111)
I20250115 02:53:12.071600 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:12.072858 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:12.080509 9171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } attempt: 1
W20250115 02:53:12.085408 9171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:42199: Network error: Client connection negotiation failed: client connection to 127.2.36.124:42199: connect: Connection refused (error 111)
I20250115 02:53:12.109720 9237 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } attempt: 1
I20250115 02:53:12.114249 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:42199
I20250115 02:53:12.114360 9300 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:42199 every 8 connection(s)
W20250115 02:53:12.114470 9237 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:42199: Network error: Client connection negotiation failed: client connection to 127.2.36.124:42199: connect: Connection refused (error 111)
I20250115 02:53:12.117604 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:53:12.118465 9301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:12.119999 9171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } attempt: 2
I20250115 02:53:12.125694 9301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:12.136073 9171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891: Bootstrap starting.
I20250115 02:53:12.136816 9301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:12.142050 9171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:12.146898 9301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:12.147617 9171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891: No bootstrap required, opened a new log
I20250115 02:53:12.150676 9171 raft_consensus.cc:357] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } }
I20250115 02:53:12.151293 9171 raft_consensus.cc:383] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:12.151541 9171 raft_consensus.cc:738] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0280d4ae264c289de9481c1909c891, State: Initialized, Role: FOLLOWER
I20250115 02:53:12.152176 9171 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } }
I20250115 02:53:12.154381 9306 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } } }
I20250115 02:53:12.155179 9306 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:12.156010 9171 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:12.164188 9301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa: Bootstrap starting.
I20250115 02:53:12.167641 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:53:12.171036 9301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa: Neither blocks nor log segments found. Creating new log.
W20250115 02:53:12.172248 9317 catalog_manager.cc:1559] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:12.172578 9317 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:12.175550 9301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa: No bootstrap required, opened a new log
I20250115 02:53:12.177716 9301 raft_consensus.cc:357] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } }
I20250115 02:53:12.178251 9301 raft_consensus.cc:383] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:12.178442 9301 raft_consensus.cc:738] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40adcf7887a74e9abc6cf7e462547caa, State: Initialized, Role: FOLLOWER
I20250115 02:53:12.178977 9301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } }
I20250115 02:53:12.180759 9319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } } }
I20250115 02:53:12.181303 9319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:12.181946 9301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:12.182215 9237 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } attempt: 2
I20250115 02:53:12.197690 9237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba: Bootstrap starting.
W20250115 02:53:12.198397 9330 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:12.198704 9330 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:12.202658 9237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:12.207070 9237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba: No bootstrap required, opened a new log
I20250115 02:53:12.209156 9237 raft_consensus.cc:357] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } }
I20250115 02:53:12.209750 9237 raft_consensus.cc:383] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:12.209969 9237 raft_consensus.cc:738] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a612a825117d48bca9f5818edf5611ba, State: Initialized, Role: FOLLOWER
I20250115 02:53:12.210490 9237 consensus_queue.cc:260] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } }
I20250115 02:53:12.212265 9332 sys_catalog.cc:455] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } } }
I20250115 02:53:12.212873 9332 sys_catalog.cc:458] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:12.213495 9237 sys_catalog.cc:564] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:12.223884 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:53:12.225761 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:12.228209 9343 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:12.228619 9343 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:53:12.233639 9344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:12.234297 9345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:12.236601 9347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:12.237820 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:12.239312 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:12.239545 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:12.239732 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909592239715 us; error 0 us; skew 500 ppm
I20250115 02:53:12.240263 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:12.242821 2193 webserver.cc:458] Webserver started at http://127.2.36.65:46167/ using document root <none> and password file <none>
I20250115 02:53:12.243274 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:12.243453 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:12.243686 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:12.244809 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "c3eb0d0d4d564259bfd93000c2610ac5"
format_stamp: "Formatted at 2025-01-15 02:53:12 on dist-test-slave-1lv2"
I20250115 02:53:12.249418 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250115 02:53:12.252997 9352 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:12.254009 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250115 02:53:12.254318 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "c3eb0d0d4d564259bfd93000c2610ac5"
format_stamp: "Formatted at 2025-01-15 02:53:12 on dist-test-slave-1lv2"
I20250115 02:53:12.254586 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:12.267469 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:12.268694 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:12.270143 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:12.272511 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:12.272711 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:12.272949 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:12.273104 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:12.299655 9319 raft_consensus.cc:491] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:12.300298 9319 raft_consensus.cc:513] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } }
I20250115 02:53:12.303711 9319 leader_election.cc:290] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb0280d4ae264c289de9481c1909c891 (127.2.36.126:40435), a612a825117d48bca9f5818edf5611ba (127.2.36.125:43831)
I20250115 02:53:12.303949 9146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40adcf7887a74e9abc6cf7e462547caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0280d4ae264c289de9481c1909c891" is_pre_election: true
I20250115 02:53:12.304283 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40adcf7887a74e9abc6cf7e462547caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a612a825117d48bca9f5818edf5611ba" is_pre_election: true
I20250115 02:53:12.304975 9146 raft_consensus.cc:2463] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40adcf7887a74e9abc6cf7e462547caa in term 0.
I20250115 02:53:12.305234 9211 raft_consensus.cc:2463] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40adcf7887a74e9abc6cf7e462547caa in term 0.
I20250115 02:53:12.306500 9250 leader_election.cc:304] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40adcf7887a74e9abc6cf7e462547caa, cb0280d4ae264c289de9481c1909c891; no voters:
I20250115 02:53:12.307448 9319 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:12.307854 9319 raft_consensus.cc:491] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:12.308211 9319 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:12.316025 9319 raft_consensus.cc:513] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } }
I20250115 02:53:12.319032 9146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40adcf7887a74e9abc6cf7e462547caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0280d4ae264c289de9481c1909c891"
I20250115 02:53:12.319208 9211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40adcf7887a74e9abc6cf7e462547caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a612a825117d48bca9f5818edf5611ba"
I20250115 02:53:12.319628 9146 raft_consensus.cc:3054] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:12.319904 9211 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:12.323766 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:38503
I20250115 02:53:12.324050 9414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:38503 every 8 connection(s)
I20250115 02:53:12.326110 9146 raft_consensus.cc:2463] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40adcf7887a74e9abc6cf7e462547caa in term 1.
I20250115 02:53:12.327205 9250 leader_election.cc:304] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40adcf7887a74e9abc6cf7e462547caa, cb0280d4ae264c289de9481c1909c891; no voters:
I20250115 02:53:12.327652 9211 raft_consensus.cc:2463] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40adcf7887a74e9abc6cf7e462547caa in term 1.
I20250115 02:53:12.328874 9416 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:12.329049 9319 leader_election.cc:290] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [CANDIDATE]: Term 1 election: Requested vote from peers cb0280d4ae264c289de9481c1909c891 (127.2.36.126:40435), a612a825117d48bca9f5818edf5611ba (127.2.36.125:43831)
I20250115 02:53:12.330181 9416 raft_consensus.cc:695] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 1 LEADER]: Becoming Leader. State: Replica: 40adcf7887a74e9abc6cf7e462547caa, State: Running, Role: LEADER
I20250115 02:53:12.331018 9416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } }
I20250115 02:53:12.346216 9319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40adcf7887a74e9abc6cf7e462547caa. Latest consensus state: current_term: 1 leader_uuid: "40adcf7887a74e9abc6cf7e462547caa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } } }
I20250115 02:53:12.347139 9319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:12.348677 9424 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:12.351984 9415 heartbeater.cc:346] Connected to a master server at 127.2.36.124:42199
I20250115 02:53:12.352406 9415 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:12.353286 9415 heartbeater.cc:509] Master 127.2.36.124:42199 requested a full tablet report, sending...
I20250115 02:53:12.355778 9266 ts_manager.cc:194] Registered new tserver with Master: c3eb0d0d4d564259bfd93000c2610ac5 (127.2.36.65:38503)
I20250115 02:53:12.356869 9424 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:12.358459 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:12.363085 9418 heartbeater.cc:346] Connected to a master server at 127.2.36.125:43831
I20250115 02:53:12.363523 9418 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:12.364511 9418 heartbeater.cc:509] Master 127.2.36.125:43831 requested a full tablet report, sending...
I20250115 02:53:12.368057 9201 ts_manager.cc:194] Registered new tserver with Master: c3eb0d0d4d564259bfd93000c2610ac5 (127.2.36.65:38503)
I20250115 02:53:12.368173 9419 heartbeater.cc:346] Connected to a master server at 127.2.36.126:40435
I20250115 02:53:12.368654 9419 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:12.369550 9419 heartbeater.cc:509] Master 127.2.36.126:40435 requested a full tablet report, sending...
I20250115 02:53:12.372503 9136 ts_manager.cc:194] Registered new tserver with Master: c3eb0d0d4d564259bfd93000c2610ac5 (127.2.36.65:38503)
W20250115 02:53:12.374914 9426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:12.375691 9146 raft_consensus.cc:1270] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [term 1 FOLLOWER]: Refusing update from remote peer 40adcf7887a74e9abc6cf7e462547caa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250115 02:53:12.378101 9427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:12.377951 9319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:12.375703 9211 raft_consensus.cc:1270] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [term 1 FOLLOWER]: Refusing update from remote peer 40adcf7887a74e9abc6cf7e462547caa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:12.382252 9319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 02:53:12.386324 9429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:12.386790 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:12.388533 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:12.388808 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:12.389123 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909592389102 us; error 0 us; skew 500 ppm
I20250115 02:53:12.389921 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:12.393347 9306 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40adcf7887a74e9abc6cf7e462547caa. Latest consensus state: current_term: 1 leader_uuid: "40adcf7887a74e9abc6cf7e462547caa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } } }
I20250115 02:53:12.394109 9306 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:12.395373 2193 webserver.cc:458] Webserver started at http://127.2.36.66:42051/ using document root <none> and password file <none>
I20250115 02:53:12.396080 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:12.396308 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:12.396622 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:12.398053 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "e82ae4d834cf40c0a55cc6d56563b15c"
format_stamp: "Formatted at 2025-01-15 02:53:12 on dist-test-slave-1lv2"
I20250115 02:53:12.400322 9431 mvcc.cc:204] Tried to move back new op lower bound from 7114381690345963520 to 7114381690197426176. Current Snapshot: MvccSnapshot[applied={T|T < 7114381690345963520}]
I20250115 02:53:12.406363 9306 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40adcf7887a74e9abc6cf7e462547caa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } } }
I20250115 02:53:12.407155 9306 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:12.409190 9416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40adcf7887a74e9abc6cf7e462547caa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } } }
I20250115 02:53:12.410004 9416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:12.408830 9424 catalog_manager.cc:1348] Generated new cluster ID: 39d4d8f57a5544ce93c2539ad60c6e32
I20250115 02:53:12.410830 9424 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:12.412694 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.014s user 0.008s sys 0.007s
I20250115 02:53:12.413232 9332 sys_catalog.cc:455] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40adcf7887a74e9abc6cf7e462547caa. Latest consensus state: current_term: 1 leader_uuid: "40adcf7887a74e9abc6cf7e462547caa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } } }
I20250115 02:53:12.413802 9332 sys_catalog.cc:458] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:12.414991 9416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40adcf7887a74e9abc6cf7e462547caa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } } }
I20250115 02:53:12.415625 9416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:12.418967 9332 sys_catalog.cc:455] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40adcf7887a74e9abc6cf7e462547caa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0280d4ae264c289de9481c1909c891" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40435 } } peers { permanent_uuid: "a612a825117d48bca9f5818edf5611ba" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43831 } } peers { permanent_uuid: "40adcf7887a74e9abc6cf7e462547caa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 42199 } } }
I20250115 02:53:12.419668 9332 sys_catalog.cc:458] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:12.422571 9441 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:12.423480 2193 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.001s sys 0.004s
I20250115 02:53:12.423899 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "e82ae4d834cf40c0a55cc6d56563b15c"
format_stamp: "Formatted at 2025-01-15 02:53:12 on dist-test-slave-1lv2"
I20250115 02:53:12.424289 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:12.456516 9424 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:53:12.457939 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:12.458339 9424 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:12.459869 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:12.461761 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:12.464483 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:12.464687 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:12.464946 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:12.465103 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:12.484838 9424 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa: Generated new TSK 0
I20250115 02:53:12.485569 9424 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:12.492959 9266 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53242
I20250115 02:53:12.506668 9415 heartbeater.cc:501] Master 127.2.36.124:42199 was elected leader, sending a full tablet report...
I20250115 02:53:12.532063 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:40729
I20250115 02:53:12.532161 9503 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:40729 every 8 connection(s)
I20250115 02:53:12.548709 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:12.557044 9504 heartbeater.cc:346] Connected to a master server at 127.2.36.124:42199
I20250115 02:53:12.557507 9504 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:12.558414 9504 heartbeater.cc:509] Master 127.2.36.124:42199 requested a full tablet report, sending...
I20250115 02:53:12.560742 9266 ts_manager.cc:194] Registered new tserver with Master: e82ae4d834cf40c0a55cc6d56563b15c (127.2.36.66:40729)
I20250115 02:53:12.562551 9266 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53244
W20250115 02:53:12.563652 9511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:12.565479 9505 heartbeater.cc:346] Connected to a master server at 127.2.36.125:43831
I20250115 02:53:12.565836 9505 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:12.566414 9505 heartbeater.cc:509] Master 127.2.36.125:43831 requested a full tablet report, sending...
I20250115 02:53:12.568759 9507 heartbeater.cc:346] Connected to a master server at 127.2.36.126:40435
I20250115 02:53:12.569128 9507 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:12.569150 9201 ts_manager.cc:194] Registered new tserver with Master: e82ae4d834cf40c0a55cc6d56563b15c (127.2.36.66:40729)
I20250115 02:53:12.569808 9507 heartbeater.cc:509] Master 127.2.36.126:40435 requested a full tablet report, sending...
W20250115 02:53:12.570382 9512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:12.572942 9136 ts_manager.cc:194] Registered new tserver with Master: e82ae4d834cf40c0a55cc6d56563b15c (127.2.36.66:40729)
I20250115 02:53:12.575731 9504 heartbeater.cc:501] Master 127.2.36.124:42199 was elected leader, sending a full tablet report...
I20250115 02:53:12.576869 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:12.577876 9514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:12.579102 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:12.579478 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:12.579727 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909592579706 us; error 0 us; skew 500 ppm
I20250115 02:53:12.580248 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:12.583067 2193 webserver.cc:458] Webserver started at http://127.2.36.67:44239/ using document root <none> and password file <none>
I20250115 02:53:12.583596 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:12.583806 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:12.584141 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:12.585189 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "f7f70cf30a584319b48bcfc106584030"
format_stamp: "Formatted at 2025-01-15 02:53:12 on dist-test-slave-1lv2"
I20250115 02:53:12.590004 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250115 02:53:12.594327 9519 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:12.595116 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:12.595430 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "f7f70cf30a584319b48bcfc106584030"
format_stamp: "Formatted at 2025-01-15 02:53:12 on dist-test-slave-1lv2"
I20250115 02:53:12.595767 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:12.609702 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:12.611449 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:12.613476 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:12.616246 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:12.616497 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:12.616756 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:12.616956 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250115 02:53:12.693475 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:42209
I20250115 02:53:12.693564 9581 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:42209 every 8 connection(s)
I20250115 02:53:12.723981 9583 heartbeater.cc:346] Connected to a master server at 127.2.36.125:43831
I20250115 02:53:12.724455 9583 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:12.725302 9583 heartbeater.cc:509] Master 127.2.36.125:43831 requested a full tablet report, sending...
I20250115 02:53:12.728104 9201 ts_manager.cc:194] Registered new tserver with Master: f7f70cf30a584319b48bcfc106584030 (127.2.36.67:42209)
I20250115 02:53:12.732656 9582 heartbeater.cc:346] Connected to a master server at 127.2.36.124:42199
I20250115 02:53:12.733088 9582 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:12.733968 9582 heartbeater.cc:509] Master 127.2.36.124:42199 requested a full tablet report, sending...
I20250115 02:53:12.736436 9264 ts_manager.cc:194] Registered new tserver with Master: f7f70cf30a584319b48bcfc106584030 (127.2.36.67:42209)
I20250115 02:53:12.738261 9264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53254
I20250115 02:53:12.747064 9585 heartbeater.cc:346] Connected to a master server at 127.2.36.126:40435
I20250115 02:53:12.747416 9585 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:12.747957 9585 heartbeater.cc:509] Master 127.2.36.126:40435 requested a full tablet report, sending...
I20250115 02:53:12.750715 9136 ts_manager.cc:194] Registered new tserver with Master: f7f70cf30a584319b48bcfc106584030 (127.2.36.67:42209)
I20250115 02:53:12.751396 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007155263s
I20250115 02:53:12.754345 9582 heartbeater.cc:501] Master 127.2.36.124:42199 was elected leader, sending a full tablet report...
I20250115 02:53:12.807381 9264 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53106:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:53:12.810335 9264 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:53:12.919925 9468 tablet_service.cc:1467] Processing CreateTablet for tablet 3084ca66d45b44ec94b89fd9c1a5915f (DEFAULT_TABLE table=ksck-test-table [id=162afcc3c267466fa3150d09466a6319]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:12.921502 9468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3084ca66d45b44ec94b89fd9c1a5915f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:12.923797 9467 tablet_service.cc:1467] Processing CreateTablet for tablet a22c0689a2ad4a878cbca4cdb64c71a3 (DEFAULT_TABLE table=ksck-test-table [id=162afcc3c267466fa3150d09466a6319]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:12.925303 9467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a22c0689a2ad4a878cbca4cdb64c71a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:12.927764 9469 tablet_service.cc:1467] Processing CreateTablet for tablet 595b018a475d4216988e090bbc76965e (DEFAULT_TABLE table=ksck-test-table [id=162afcc3c267466fa3150d09466a6319]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:12.929840 9469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 595b018a475d4216988e090bbc76965e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:12.970227 9547 tablet_service.cc:1467] Processing CreateTablet for tablet 595b018a475d4216988e090bbc76965e (DEFAULT_TABLE table=ksck-test-table [id=162afcc3c267466fa3150d09466a6319]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:12.970817 9546 tablet_service.cc:1467] Processing CreateTablet for tablet 3084ca66d45b44ec94b89fd9c1a5915f (DEFAULT_TABLE table=ksck-test-table [id=162afcc3c267466fa3150d09466a6319]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:12.972258 9546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3084ca66d45b44ec94b89fd9c1a5915f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:12.972955 9603 tablet_bootstrap.cc:492] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c: Bootstrap starting.
I20250115 02:53:12.974531 9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 595b018a475d4216988e090bbc76965e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:12.970712 9545 tablet_service.cc:1467] Processing CreateTablet for tablet a22c0689a2ad4a878cbca4cdb64c71a3 (DEFAULT_TABLE table=ksck-test-table [id=162afcc3c267466fa3150d09466a6319]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:12.976001 9545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a22c0689a2ad4a878cbca4cdb64c71a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:12.979619 9603 tablet_bootstrap.cc:654] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:12.993477 9603 tablet_bootstrap.cc:492] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c: No bootstrap required, opened a new log
I20250115 02:53:12.993985 9603 ts_tablet_manager.cc:1397] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.005s
I20250115 02:53:13.005298 9603 raft_consensus.cc:357] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.006081 9603 raft_consensus.cc:383] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:13.006376 9603 raft_consensus.cc:738] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82ae4d834cf40c0a55cc6d56563b15c, State: Initialized, Role: FOLLOWER
I20250115 02:53:13.007325 9603 consensus_queue.cc:260] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.011585 9605 tablet_bootstrap.cc:492] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030: Bootstrap starting.
I20250115 02:53:13.019032 9380 tablet_service.cc:1467] Processing CreateTablet for tablet 595b018a475d4216988e090bbc76965e (DEFAULT_TABLE table=ksck-test-table [id=162afcc3c267466fa3150d09466a6319]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:13.020512 9380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 595b018a475d4216988e090bbc76965e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:13.022583 9605 tablet_bootstrap.cc:654] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:13.023797 9379 tablet_service.cc:1467] Processing CreateTablet for tablet 3084ca66d45b44ec94b89fd9c1a5915f (DEFAULT_TABLE table=ksck-test-table [id=162afcc3c267466fa3150d09466a6319]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:13.025151 9379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3084ca66d45b44ec94b89fd9c1a5915f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:13.027784 9378 tablet_service.cc:1467] Processing CreateTablet for tablet a22c0689a2ad4a878cbca4cdb64c71a3 (DEFAULT_TABLE table=ksck-test-table [id=162afcc3c267466fa3150d09466a6319]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:13.029086 9378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a22c0689a2ad4a878cbca4cdb64c71a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:13.057054 9603 ts_tablet_manager.cc:1428] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c: Time spent starting tablet: real 0.063s user 0.026s sys 0.002s
I20250115 02:53:13.058326 9603 tablet_bootstrap.cc:492] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c: Bootstrap starting.
I20250115 02:53:13.073088 9603 tablet_bootstrap.cc:654] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:13.089812 9605 tablet_bootstrap.cc:492] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030: No bootstrap required, opened a new log
I20250115 02:53:13.091547 9606 raft_consensus.cc:491] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:13.092904 9606 raft_consensus.cc:513] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.095252 9606 leader_election.cc:290] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3eb0d0d4d564259bfd93000c2610ac5 (127.2.36.65:38503), f7f70cf30a584319b48bcfc106584030 (127.2.36.67:42209)
I20250115 02:53:13.093516 9609 tablet_bootstrap.cc:492] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5: Bootstrap starting.
I20250115 02:53:13.098744 9603 tablet_bootstrap.cc:492] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c: No bootstrap required, opened a new log
I20250115 02:53:13.099298 9603 ts_tablet_manager.cc:1397] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.005s
I20250115 02:53:13.099267 9605 ts_tablet_manager.cc:1397] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030: Time spent bootstrapping tablet: real 0.089s user 0.010s sys 0.016s
I20250115 02:53:13.102382 9603 raft_consensus.cc:357] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.103163 9603 raft_consensus.cc:383] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:13.103456 9603 raft_consensus.cc:738] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82ae4d834cf40c0a55cc6d56563b15c, State: Initialized, Role: FOLLOWER
I20250115 02:53:13.104135 9603 consensus_queue.cc:260] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.106438 9603 ts_tablet_manager.cc:1428] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250115 02:53:13.106299 9605 raft_consensus.cc:357] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } } peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } }
I20250115 02:53:13.108732 9609 tablet_bootstrap.cc:654] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:13.109079 9605 raft_consensus.cc:383] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:13.109511 9605 raft_consensus.cc:738] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f70cf30a584319b48bcfc106584030, State: Initialized, Role: FOLLOWER
I20250115 02:53:13.112334 9605 consensus_queue.cc:260] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } } peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } }
I20250115 02:53:13.117275 9603 tablet_bootstrap.cc:492] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c: Bootstrap starting.
I20250115 02:53:13.123615 9609 tablet_bootstrap.cc:492] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5: No bootstrap required, opened a new log
I20250115 02:53:13.124138 9609 ts_tablet_manager.cc:1397] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.006s
I20250115 02:53:13.124380 9603 tablet_bootstrap.cc:654] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:13.127300 9609 raft_consensus.cc:357] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.128155 9609 raft_consensus.cc:383] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:13.128470 9609 raft_consensus.cc:738] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3eb0d0d4d564259bfd93000c2610ac5, State: Initialized, Role: FOLLOWER
I20250115 02:53:13.129292 9609 consensus_queue.cc:260] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.141098 9609 ts_tablet_manager.cc:1428] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5: Time spent starting tablet: real 0.017s user 0.005s sys 0.005s
I20250115 02:53:13.145035 9609 tablet_bootstrap.cc:492] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5: Bootstrap starting.
I20250115 02:53:13.152281 9603 tablet_bootstrap.cc:492] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c: No bootstrap required, opened a new log
I20250115 02:53:13.152930 9603 ts_tablet_manager.cc:1397] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.014s
I20250115 02:53:13.156505 9609 tablet_bootstrap.cc:654] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:13.159168 9603 raft_consensus.cc:357] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } } peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } }
I20250115 02:53:13.159970 9603 raft_consensus.cc:383] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:13.160274 9603 raft_consensus.cc:738] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82ae4d834cf40c0a55cc6d56563b15c, State: Initialized, Role: FOLLOWER
I20250115 02:53:13.161000 9603 consensus_queue.cc:260] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } } peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } }
I20250115 02:53:13.164345 9603 ts_tablet_manager.cc:1428] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250115 02:53:13.169489 9609 tablet_bootstrap.cc:492] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5: No bootstrap required, opened a new log
I20250115 02:53:13.170049 9609 ts_tablet_manager.cc:1397] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.002s
I20250115 02:53:13.172905 9609 raft_consensus.cc:357] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } } peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } }
I20250115 02:53:13.173650 9609 raft_consensus.cc:383] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:13.173921 9609 raft_consensus.cc:738] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3eb0d0d4d564259bfd93000c2610ac5, State: Initialized, Role: FOLLOWER
I20250115 02:53:13.174566 9609 consensus_queue.cc:260] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } } peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } }
I20250115 02:53:13.177189 9609 ts_tablet_manager.cc:1428] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250115 02:53:13.178267 9609 tablet_bootstrap.cc:492] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5: Bootstrap starting.
I20250115 02:53:13.178253 9317 catalog_manager.cc:1260] Loaded cluster ID: 39d4d8f57a5544ce93c2539ad60c6e32
I20250115 02:53:13.179198 9317 catalog_manager.cc:1553] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891: loading cluster ID for follower catalog manager: success
I20250115 02:53:13.185401 9609 tablet_bootstrap.cc:654] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:13.201187 9317 catalog_manager.cc:1575] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891: acquiring CA information for follower catalog manager: success
I20250115 02:53:13.213330 9605 ts_tablet_manager.cc:1428] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030: Time spent starting tablet: real 0.113s user 0.018s sys 0.019s
I20250115 02:53:13.229395 9609 tablet_bootstrap.cc:492] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5: No bootstrap required, opened a new log
I20250115 02:53:13.230048 9609 ts_tablet_manager.cc:1397] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5: Time spent bootstrapping tablet: real 0.052s user 0.030s sys 0.019s
I20250115 02:53:13.233147 9609 raft_consensus.cc:357] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.234026 9609 raft_consensus.cc:383] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:13.234349 9609 raft_consensus.cc:738] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3eb0d0d4d564259bfd93000c2610ac5, State: Initialized, Role: FOLLOWER
I20250115 02:53:13.235234 9609 consensus_queue.cc:260] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.237808 9609 ts_tablet_manager.cc:1428] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20250115 02:53:13.256902 9343 catalog_manager.cc:1260] Loaded cluster ID: 39d4d8f57a5544ce93c2539ad60c6e32
I20250115 02:53:13.256703 9390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a22c0689a2ad4a878cbca4cdb64c71a3" candidate_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" is_pre_election: true
I20250115 02:53:13.257232 9343 catalog_manager.cc:1553] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba: loading cluster ID for follower catalog manager: success
I20250115 02:53:13.257472 9390 raft_consensus.cc:2463] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e82ae4d834cf40c0a55cc6d56563b15c in term 0.
I20250115 02:53:13.258571 9442 leader_election.cc:304] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3eb0d0d4d564259bfd93000c2610ac5, e82ae4d834cf40c0a55cc6d56563b15c; no voters:
I20250115 02:53:13.259487 9606 raft_consensus.cc:2798] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:13.259912 9606 raft_consensus.cc:491] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:13.260255 9606 raft_consensus.cc:3054] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:13.273761 9606 raft_consensus.cc:513] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.276173 9557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a22c0689a2ad4a878cbca4cdb64c71a3" candidate_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f70cf30a584319b48bcfc106584030" is_pre_election: true
I20250115 02:53:13.277266 9606 leader_election.cc:290] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [CANDIDATE]: Term 1 election: Requested vote from peers c3eb0d0d4d564259bfd93000c2610ac5 (127.2.36.65:38503), f7f70cf30a584319b48bcfc106584030 (127.2.36.67:42209)
I20250115 02:53:13.277882 9390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a22c0689a2ad4a878cbca4cdb64c71a3" candidate_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3eb0d0d4d564259bfd93000c2610ac5"
I20250115 02:53:13.278533 9390 raft_consensus.cc:3054] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:13.287214 9557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a22c0689a2ad4a878cbca4cdb64c71a3" candidate_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f70cf30a584319b48bcfc106584030"
W20250115 02:53:13.286201 9445 leader_election.cc:343] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7f70cf30a584319b48bcfc106584030 (127.2.36.67:42209): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:13.290983 9390 raft_consensus.cc:2463] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e82ae4d834cf40c0a55cc6d56563b15c in term 1.
I20250115 02:53:13.293440 9605 tablet_bootstrap.cc:492] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030: Bootstrap starting.
I20250115 02:53:13.297857 9317 catalog_manager.cc:1603] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250115 02:53:13.299793 9509 tablet.cc:2367] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250115 02:53:13.300128 9605 tablet_bootstrap.cc:654] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:13.300649 9343 catalog_manager.cc:1575] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba: acquiring CA information for follower catalog manager: success
I20250115 02:53:13.303114 9442 leader_election.cc:304] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3eb0d0d4d564259bfd93000c2610ac5, e82ae4d834cf40c0a55cc6d56563b15c; no voters:
I20250115 02:53:13.304562 9606 raft_consensus.cc:2798] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:13.306712 9343 catalog_manager.cc:1603] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250115 02:53:13.320210 9445 leader_election.cc:343] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7f70cf30a584319b48bcfc106584030 (127.2.36.67:42209): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:13.325659 9606 raft_consensus.cc:695] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 1 LEADER]: Becoming Leader. State: Replica: e82ae4d834cf40c0a55cc6d56563b15c, State: Running, Role: LEADER
I20250115 02:53:13.326821 9605 tablet_bootstrap.cc:492] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030: No bootstrap required, opened a new log
I20250115 02:53:13.327512 9605 ts_tablet_manager.cc:1397] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.004s
I20250115 02:53:13.326900 9606 consensus_queue.cc:237] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.331089 9605 raft_consensus.cc:357] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.331859 9605 raft_consensus.cc:383] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:13.332144 9605 raft_consensus.cc:738] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f70cf30a584319b48bcfc106584030, State: Initialized, Role: FOLLOWER
I20250115 02:53:13.332988 9605 consensus_queue.cc:260] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.335819 9605 ts_tablet_manager.cc:1428] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20250115 02:53:13.336894 9605 tablet_bootstrap.cc:492] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030: Bootstrap starting.
I20250115 02:53:13.342780 9605 tablet_bootstrap.cc:654] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:13.343174 9263 catalog_manager.cc:5526] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c reported cstate change: term changed from 0 to 1, leader changed from <none> to e82ae4d834cf40c0a55cc6d56563b15c (127.2.36.66). New cstate: current_term: 1 leader_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:13.361193 9605 tablet_bootstrap.cc:492] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030: No bootstrap required, opened a new log
I20250115 02:53:13.361688 9605 ts_tablet_manager.cc:1397] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.008s
I20250115 02:53:13.366590 9615 raft_consensus.cc:491] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:13.367128 9615 raft_consensus.cc:513] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.367451 9605 raft_consensus.cc:357] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.368177 9605 raft_consensus.cc:383] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:13.368501 9605 raft_consensus.cc:738] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f70cf30a584319b48bcfc106584030, State: Initialized, Role: FOLLOWER
I20250115 02:53:13.382095 9615 leader_election.cc:290] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e82ae4d834cf40c0a55cc6d56563b15c (127.2.36.66:40729), f7f70cf30a584319b48bcfc106584030 (127.2.36.67:42209)
I20250115 02:53:13.369179 9605 consensus_queue.cc:260] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.384444 9605 ts_tablet_manager.cc:1428] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030: Time spent starting tablet: real 0.022s user 0.014s sys 0.006s
I20250115 02:53:13.402251 9613 raft_consensus.cc:491] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:13.403044 9557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3084ca66d45b44ec94b89fd9c1a5915f" candidate_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f70cf30a584319b48bcfc106584030" is_pre_election: true
I20250115 02:53:13.403625 9557 raft_consensus.cc:2463] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3eb0d0d4d564259bfd93000c2610ac5 in term 0.
I20250115 02:53:13.402802 9613 raft_consensus.cc:513] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } } peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } }
I20250115 02:53:13.410305 9613 leader_election.cc:290] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e82ae4d834cf40c0a55cc6d56563b15c (127.2.36.66:40729), c3eb0d0d4d564259bfd93000c2610ac5 (127.2.36.65:38503)
I20250115 02:53:13.415895 9356 leader_election.cc:304] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3eb0d0d4d564259bfd93000c2610ac5, f7f70cf30a584319b48bcfc106584030; no voters:
I20250115 02:53:13.417420 9615 raft_consensus.cc:2798] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:13.417824 9615 raft_consensus.cc:491] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:13.417445 9479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3084ca66d45b44ec94b89fd9c1a5915f" candidate_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" is_pre_election: true
I20250115 02:53:13.422236 9615 raft_consensus.cc:3054] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:13.422538 9479 raft_consensus.cc:2463] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3eb0d0d4d564259bfd93000c2610ac5 in term 0.
I20250115 02:53:13.432535 9615 raft_consensus.cc:513] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.435370 9479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3084ca66d45b44ec94b89fd9c1a5915f" candidate_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82ae4d834cf40c0a55cc6d56563b15c"
I20250115 02:53:13.436028 9479 raft_consensus.cc:3054] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:13.438061 9615 leader_election.cc:290] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [CANDIDATE]: Term 1 election: Requested vote from peers e82ae4d834cf40c0a55cc6d56563b15c (127.2.36.66:40729), f7f70cf30a584319b48bcfc106584030 (127.2.36.67:42209)
I20250115 02:53:13.443346 9479 raft_consensus.cc:2463] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3eb0d0d4d564259bfd93000c2610ac5 in term 1.
I20250115 02:53:13.439345 9557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3084ca66d45b44ec94b89fd9c1a5915f" candidate_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f70cf30a584319b48bcfc106584030"
I20250115 02:53:13.444262 9557 raft_consensus.cc:3054] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:13.452200 9557 raft_consensus.cc:2463] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3eb0d0d4d564259bfd93000c2610ac5 in term 1.
I20250115 02:53:13.452788 9355 leader_election.cc:304] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3eb0d0d4d564259bfd93000c2610ac5, e82ae4d834cf40c0a55cc6d56563b15c; no voters:
I20250115 02:53:13.454701 9478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595b018a475d4216988e090bbc76965e" candidate_uuid: "f7f70cf30a584319b48bcfc106584030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" is_pre_election: true
I20250115 02:53:13.455497 9478 raft_consensus.cc:2463] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7f70cf30a584319b48bcfc106584030 in term 0.
I20250115 02:53:13.455476 9615 raft_consensus.cc:2798] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:13.460489 9615 raft_consensus.cc:695] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 1 LEADER]: Becoming Leader. State: Replica: c3eb0d0d4d564259bfd93000c2610ac5, State: Running, Role: LEADER
I20250115 02:53:13.456938 9522 leader_election.cc:304] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e82ae4d834cf40c0a55cc6d56563b15c, f7f70cf30a584319b48bcfc106584030; no voters:
I20250115 02:53:13.461035 9390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595b018a475d4216988e090bbc76965e" candidate_uuid: "f7f70cf30a584319b48bcfc106584030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" is_pre_election: true
I20250115 02:53:13.462010 9613 raft_consensus.cc:2798] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:13.462568 9613 raft_consensus.cc:491] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:13.461483 9615 consensus_queue.cc:237] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } }
I20250115 02:53:13.462960 9613 raft_consensus.cc:3054] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:13.462240 9390 raft_consensus.cc:2463] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7f70cf30a584319b48bcfc106584030 in term 0.
I20250115 02:53:13.476698 9613 raft_consensus.cc:513] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } } peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } }
I20250115 02:53:13.479318 9613 leader_election.cc:290] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [CANDIDATE]: Term 1 election: Requested vote from peers e82ae4d834cf40c0a55cc6d56563b15c (127.2.36.66:40729), c3eb0d0d4d564259bfd93000c2610ac5 (127.2.36.65:38503)
I20250115 02:53:13.479715 9390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595b018a475d4216988e090bbc76965e" candidate_uuid: "f7f70cf30a584319b48bcfc106584030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3eb0d0d4d564259bfd93000c2610ac5"
I20250115 02:53:13.480397 9390 raft_consensus.cc:3054] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:13.482641 9479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595b018a475d4216988e090bbc76965e" candidate_uuid: "f7f70cf30a584319b48bcfc106584030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82ae4d834cf40c0a55cc6d56563b15c"
I20250115 02:53:13.483342 9479 raft_consensus.cc:3054] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:13.492624 9479 raft_consensus.cc:2463] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f70cf30a584319b48bcfc106584030 in term 1.
I20250115 02:53:13.493264 9390 raft_consensus.cc:2463] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f70cf30a584319b48bcfc106584030 in term 1.
I20250115 02:53:13.493036 9266 catalog_manager.cc:5526] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3eb0d0d4d564259bfd93000c2610ac5 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:13.493808 9522 leader_election.cc:304] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e82ae4d834cf40c0a55cc6d56563b15c, f7f70cf30a584319b48bcfc106584030; no voters:
I20250115 02:53:13.501447 9613 raft_consensus.cc:2798] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:13.502836 9613 raft_consensus.cc:695] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 1 LEADER]: Becoming Leader. State: Replica: f7f70cf30a584319b48bcfc106584030, State: Running, Role: LEADER
I20250115 02:53:13.503705 9613 consensus_queue.cc:237] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } } peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } }
I20250115 02:53:13.529551 9263 catalog_manager.cc:5526] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f70cf30a584319b48bcfc106584030 (127.2.36.67). New cstate: current_term: 1 leader_uuid: "f7f70cf30a584319b48bcfc106584030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ae4d834cf40c0a55cc6d56563b15c" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 40729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:13.679701 9263 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53106:
table { table_name: "ksck-test-table" } modify_external_catalogs: true
I20250115 02:53:13.680423 9263 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53106:
table { table_name: "ksck-test-table" } modify_external_catalogs: true
I20250115 02:53:13.709554 9263 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa: Sending DeleteTablet for 3 replicas of tablet 595b018a475d4216988e090bbc76965e
I20250115 02:53:13.711331 9263 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa: Sending DeleteTablet for 3 replicas of tablet 3084ca66d45b44ec94b89fd9c1a5915f
I20250115 02:53:13.711537 9469 tablet_service.cc:1514] Processing DeleteTablet for tablet 595b018a475d4216988e090bbc76965e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 02:53:13 UTC) from {username='slave'} at 127.0.0.1:39682
I20250115 02:53:13.711841 9545 tablet_service.cc:1514] Processing DeleteTablet for tablet 595b018a475d4216988e090bbc76965e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 02:53:13 UTC) from {username='slave'} at 127.0.0.1:51404
I20250115 02:53:13.712937 9263 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa: Sending DeleteTablet for 3 replicas of tablet a22c0689a2ad4a878cbca4cdb64c71a3
I20250115 02:53:13.714111 9546 tablet_service.cc:1514] Processing DeleteTablet for tablet 3084ca66d45b44ec94b89fd9c1a5915f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 02:53:13 UTC) from {username='slave'} at 127.0.0.1:51404
I20250115 02:53:13.716642 9468 tablet_service.cc:1514] Processing DeleteTablet for tablet a22c0689a2ad4a878cbca4cdb64c71a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 02:53:13 UTC) from {username='slave'} at 127.0.0.1:39682
I20250115 02:53:13.716710 9467 tablet_service.cc:1514] Processing DeleteTablet for tablet 3084ca66d45b44ec94b89fd9c1a5915f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 02:53:13 UTC) from {username='slave'} at 127.0.0.1:39682
I20250115 02:53:13.723364 9545 tablet_service.cc:1514] Processing DeleteTablet for tablet a22c0689a2ad4a878cbca4cdb64c71a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 02:53:13 UTC) from {username='slave'} at 127.0.0.1:51404
I20250115 02:53:13.723573 9637 tablet_replica.cc:331] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c: stopping tablet replica
I20250115 02:53:13.724279 9638 tablet_replica.cc:331] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030: stopping tablet replica
I20250115 02:53:13.724771 9380 tablet_service.cc:1514] Processing DeleteTablet for tablet 3084ca66d45b44ec94b89fd9c1a5915f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 02:53:13 UTC) from {username='slave'} at 127.0.0.1:42520
I20250115 02:53:13.725184 9638 raft_consensus.cc:2238] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:13.726109 9378 tablet_service.cc:1514] Processing DeleteTablet for tablet a22c0689a2ad4a878cbca4cdb64c71a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 02:53:13 UTC) from {username='slave'} at 127.0.0.1:42520
I20250115 02:53:13.713021 9379 tablet_service.cc:1514] Processing DeleteTablet for tablet 595b018a475d4216988e090bbc76965e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 02:53:13 UTC) from {username='slave'} at 127.0.0.1:42520
I20250115 02:53:13.728420 9638 pending_rounds.cc:62] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030: Trying to abort 1 pending ops.
I20250115 02:53:13.728688 9638 pending_rounds.cc:69] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114381695000670208 op_type: NO_OP noop_request { }
I20250115 02:53:13.728854 9557 raft_consensus.cc:3054] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:13.729151 9638 raft_consensus.cc:2883] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 02:53:13.729624 9638 raft_consensus.cc:2267] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:13.734115 9637 raft_consensus.cc:2238] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:13.734771 9637 raft_consensus.cc:2267] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:13.737452 9637 ts_tablet_manager.cc:1905] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:13.742537 9638 ts_tablet_manager.cc:1905] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:13.742592 9639 tablet_replica.cc:331] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5: stopping tablet replica
I20250115 02:53:13.745118 9639 raft_consensus.cc:2238] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:13.745983 9639 pending_rounds.cc:62] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5: Trying to abort 1 pending ops.
I20250115 02:53:13.746248 9639 pending_rounds.cc:69] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114381694831329280 op_type: NO_OP noop_request { }
I20250115 02:53:13.746603 9639 raft_consensus.cc:2883] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 02:53:13.746994 9639 raft_consensus.cc:2267] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:13.751487 9639 ts_tablet_manager.cc:1905] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:13.753716 9606 consensus_queue.cc:1035] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f70cf30a584319b48bcfc106584030" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:13.768710 9638 ts_tablet_manager.cc:1918] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 02:53:13.769722 9638 log.cc:1198] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-2-root/wals/595b018a475d4216988e090bbc76965e
I20250115 02:53:13.770417 9638 ts_tablet_manager.cc:1939] T 595b018a475d4216988e090bbc76965e P f7f70cf30a584319b48bcfc106584030: Deleting consensus metadata
I20250115 02:53:13.785066 9253 catalog_manager.cc:4872] TS f7f70cf30a584319b48bcfc106584030 (127.2.36.67:42209): tablet 595b018a475d4216988e090bbc76965e (table ksck-test-table [id=162afcc3c267466fa3150d09466a6319]) successfully deleted
I20250115 02:53:13.786197 9638 tablet_replica.cc:331] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030: stopping tablet replica
I20250115 02:53:13.787230 9638 raft_consensus.cc:2238] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:13.787865 9638 raft_consensus.cc:2267] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:13.791983 9637 ts_tablet_manager.cc:1918] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 02:53:13.792423 9637 log.cc:1198] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-1-root/wals/595b018a475d4216988e090bbc76965e
I20250115 02:53:13.793107 9637 ts_tablet_manager.cc:1939] T 595b018a475d4216988e090bbc76965e P e82ae4d834cf40c0a55cc6d56563b15c: Deleting consensus metadata
I20250115 02:53:13.793140 9638 ts_tablet_manager.cc:1905] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:13.805503 9606 consensus_queue.cc:1035] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3eb0d0d4d564259bfd93000c2610ac5" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:13.812337 9252 catalog_manager.cc:4872] TS e82ae4d834cf40c0a55cc6d56563b15c (127.2.36.66:40729): tablet 595b018a475d4216988e090bbc76965e (table ksck-test-table [id=162afcc3c267466fa3150d09466a6319]) successfully deleted
I20250115 02:53:13.812541 9637 tablet_replica.cc:331] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c: stopping tablet replica
I20250115 02:53:13.813979 9637 raft_consensus.cc:2238] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:13.815250 9637 raft_consensus.cc:2267] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:13.818974 9638 ts_tablet_manager.cc:1918] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 02:53:13.819456 9638 log.cc:1198] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-2-root/wals/3084ca66d45b44ec94b89fd9c1a5915f
I20250115 02:53:13.820299 9638 ts_tablet_manager.cc:1939] T 3084ca66d45b44ec94b89fd9c1a5915f P f7f70cf30a584319b48bcfc106584030: Deleting consensus metadata
I20250115 02:53:13.821058 9637 ts_tablet_manager.cc:1905] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:13.823549 9253 catalog_manager.cc:4872] TS f7f70cf30a584319b48bcfc106584030 (127.2.36.67:42209): tablet 3084ca66d45b44ec94b89fd9c1a5915f (table ksck-test-table [id=162afcc3c267466fa3150d09466a6319]) successfully deleted
I20250115 02:53:13.824684 9638 tablet_replica.cc:331] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030: stopping tablet replica
I20250115 02:53:13.826690 9639 ts_tablet_manager.cc:1918] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 02:53:13.827275 9639 log.cc:1198] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-0-root/wals/3084ca66d45b44ec94b89fd9c1a5915f
I20250115 02:53:13.828418 9639 ts_tablet_manager.cc:1939] T 3084ca66d45b44ec94b89fd9c1a5915f P c3eb0d0d4d564259bfd93000c2610ac5: Deleting consensus metadata
I20250115 02:53:13.835072 9638 raft_consensus.cc:2238] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:13.845886 9638 raft_consensus.cc:2267] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:13.852986 9638 ts_tablet_manager.cc:1905] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:13.864557 9250 catalog_manager.cc:4872] TS c3eb0d0d4d564259bfd93000c2610ac5 (127.2.36.65:38503): tablet 3084ca66d45b44ec94b89fd9c1a5915f (table ksck-test-table [id=162afcc3c267466fa3150d09466a6319]) successfully deleted
I20250115 02:53:13.879022 9637 ts_tablet_manager.cc:1918] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 02:53:13.879611 9637 log.cc:1198] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-1-root/wals/a22c0689a2ad4a878cbca4cdb64c71a3
I20250115 02:53:13.880436 9637 ts_tablet_manager.cc:1939] T a22c0689a2ad4a878cbca4cdb64c71a3 P e82ae4d834cf40c0a55cc6d56563b15c: Deleting consensus metadata
I20250115 02:53:13.883980 9639 tablet_replica.cc:331] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5: stopping tablet replica
I20250115 02:53:13.887683 9252 catalog_manager.cc:4872] TS e82ae4d834cf40c0a55cc6d56563b15c (127.2.36.66:40729): tablet a22c0689a2ad4a878cbca4cdb64c71a3 (table ksck-test-table [id=162afcc3c267466fa3150d09466a6319]) successfully deleted
I20250115 02:53:13.888784 9637 tablet_replica.cc:331] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c: stopping tablet replica
I20250115 02:53:13.893986 9637 raft_consensus.cc:2238] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:13.894567 9637 raft_consensus.cc:2267] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:13.895344 9639 raft_consensus.cc:2238] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:13.895952 9639 raft_consensus.cc:2267] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:13.900681 9639 ts_tablet_manager.cc:1905] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:13.906574 9638 ts_tablet_manager.cc:1918] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 02:53:13.907090 9638 log.cc:1198] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-2-root/wals/a22c0689a2ad4a878cbca4cdb64c71a3
I20250115 02:53:13.908159 9638 ts_tablet_manager.cc:1939] T a22c0689a2ad4a878cbca4cdb64c71a3 P f7f70cf30a584319b48bcfc106584030: Deleting consensus metadata
I20250115 02:53:13.917099 9253 catalog_manager.cc:4872] TS f7f70cf30a584319b48bcfc106584030 (127.2.36.67:42209): tablet a22c0689a2ad4a878cbca4cdb64c71a3 (table ksck-test-table [id=162afcc3c267466fa3150d09466a6319]) successfully deleted
I20250115 02:53:13.930073 9637 ts_tablet_manager.cc:1905] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:13.967492 9637 ts_tablet_manager.cc:1918] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 02:53:13.968019 9637 log.cc:1198] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-1-root/wals/3084ca66d45b44ec94b89fd9c1a5915f
I20250115 02:53:13.968763 9637 ts_tablet_manager.cc:1939] T 3084ca66d45b44ec94b89fd9c1a5915f P e82ae4d834cf40c0a55cc6d56563b15c: Deleting consensus metadata
I20250115 02:53:13.975867 9639 ts_tablet_manager.cc:1918] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 02:53:13.976330 9639 log.cc:1198] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-0-root/wals/a22c0689a2ad4a878cbca4cdb64c71a3
I20250115 02:53:13.977075 9639 ts_tablet_manager.cc:1939] T a22c0689a2ad4a878cbca4cdb64c71a3 P c3eb0d0d4d564259bfd93000c2610ac5: Deleting consensus metadata
I20250115 02:53:13.977336 9252 catalog_manager.cc:4872] TS e82ae4d834cf40c0a55cc6d56563b15c (127.2.36.66:40729): tablet 3084ca66d45b44ec94b89fd9c1a5915f (table ksck-test-table [id=162afcc3c267466fa3150d09466a6319]) successfully deleted
I20250115 02:53:13.982059 9250 catalog_manager.cc:4872] TS c3eb0d0d4d564259bfd93000c2610ac5 (127.2.36.65:38503): tablet a22c0689a2ad4a878cbca4cdb64c71a3 (table ksck-test-table [id=162afcc3c267466fa3150d09466a6319]) successfully deleted
I20250115 02:53:13.981976 9639 tablet_replica.cc:331] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5: stopping tablet replica
I20250115 02:53:13.983515 9639 raft_consensus.cc:2238] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:13.984755 9639 raft_consensus.cc:2267] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:13.987233 9639 ts_tablet_manager.cc:1905] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:14.003767 9639 ts_tablet_manager.cc:1918] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 02:53:14.004182 9639 log.cc:1198] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNoTableCluster.1736909525896969-2193-0/minicluster-data/ts-0-root/wals/595b018a475d4216988e090bbc76965e
I20250115 02:53:14.004889 9639 ts_tablet_manager.cc:1939] T 595b018a475d4216988e090bbc76965e P c3eb0d0d4d564259bfd93000c2610ac5: Deleting consensus metadata
I20250115 02:53:14.009140 9250 catalog_manager.cc:4872] TS c3eb0d0d4d564259bfd93000c2610ac5 (127.2.36.65:38503): tablet 595b018a475d4216988e090bbc76965e (table ksck-test-table [id=162afcc3c267466fa3150d09466a6319]) successfully deleted
I20250115 02:53:14.487807 9380 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:53:14.501351 9469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:53:14.502672 9545 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:53:14.847158 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:53:14.911547 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:14.928323 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:53:14.943789 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:53:14.980101 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:14.986524 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:53:15.001487 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:53:15.026039 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:15.041669 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:53:15.053252 2193 master.cc:537] Master@127.2.36.126:40435 shutting down...
I20250115 02:53:15.072700 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:15.073226 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:15.073525 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P cb0280d4ae264c289de9481c1909c891: stopping tablet replica
I20250115 02:53:15.092242 2193 master.cc:559] Master@127.2.36.126:40435 shutdown complete.
I20250115 02:53:15.104105 2193 master.cc:537] Master@127.2.36.125:43831 shutting down...
I20250115 02:53:15.121104 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:15.121608 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:15.121891 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P a612a825117d48bca9f5818edf5611ba: stopping tablet replica
I20250115 02:53:15.141595 2193 master.cc:559] Master@127.2.36.125:43831 shutdown complete.
W20250115 02:53:15.149956 9250 consensus_peers.cc:487] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa -> Peer cb0280d4ae264c289de9481c1909c891 (127.2.36.126:40435): Couldn't send request to peer cb0280d4ae264c289de9481c1909c891. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.126:40435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:53:15.153071 2193 master.cc:537] Master@127.2.36.124:42199 shutting down...
I20250115 02:53:15.169698 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:15.170512 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:15.170858 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 40adcf7887a74e9abc6cf7e462547caa: stopping tablet replica
I20250115 02:53:15.190014 2193 master.cc:559] Master@127.2.36.124:42199 shutdown complete.
[ OK ] RemoteKsckTest.TestFilterOnNoTableCluster (3404 ms)
[ RUN ] RemoteKsckTest.TestNonMatchingTableFilter
I20250115 02:53:15.217921 2193 test_util.cc:274] Using random seed: -949088278
I20250115 02:53:15.218904 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:33631,127.2.36.125:42825,127.2.36.124:36699
I20250115 02:53:15.220140 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:15.225746 9680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:15.226346 9681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:15.227035 9683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:15.227900 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:15.228641 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:15.228834 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:15.228972 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909595228956 us; error 0 us; skew 500 ppm
I20250115 02:53:15.229476 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:15.231783 2193 webserver.cc:458] Webserver started at http://127.2.36.126:46003/ using document root <none> and password file <none>
I20250115 02:53:15.232232 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:15.232403 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:15.232637 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:15.233671 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "22aa6472c6b94725b1f14899990faf4b"
format_stamp: "Formatted at 2025-01-15 02:53:15 on dist-test-slave-1lv2"
I20250115 02:53:15.237906 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250115 02:53:15.241194 9688 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:15.241902 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:53:15.242180 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "22aa6472c6b94725b1f14899990faf4b"
format_stamp: "Formatted at 2025-01-15 02:53:15 on dist-test-slave-1lv2"
I20250115 02:53:15.242441 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:15.262955 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:15.263891 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:15.296329 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:33631
I20250115 02:53:15.296417 9739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:33631 every 8 connection(s)
I20250115 02:53:15.299870 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:15.300181 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:15.306180 9742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:15.307150 9743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:15.307817 9740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:15.307989 9745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:15.309067 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:15.310269 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:15.310520 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:15.310676 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909595310660 us; error 0 us; skew 500 ppm
I20250115 02:53:15.311254 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:15.313962 2193 webserver.cc:458] Webserver started at http://127.2.36.125:36753/ using document root <none> and password file <none>
I20250115 02:53:15.314430 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:15.314599 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:15.314864 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:15.316047 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "bc7660636b6546f5bc9c08a052af62c4"
format_stamp: "Formatted at 2025-01-15 02:53:15 on dist-test-slave-1lv2"
I20250115 02:53:15.320883 9740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:15.321058 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250115 02:53:15.325716 9753 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:15.326480 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250115 02:53:15.326802 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "bc7660636b6546f5bc9c08a052af62c4"
format_stamp: "Formatted at 2025-01-15 02:53:15 on dist-test-slave-1lv2"
W20250115 02:53:15.326920 9740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:42825: Network error: Client connection negotiation failed: client connection to 127.2.36.125:42825: connect: Connection refused (error 111)
I20250115 02:53:15.327147 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:15.346349 9740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } attempt: 1
I20250115 02:53:15.346841 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:15.347977 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250115 02:53:15.350600 9740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:42825: Network error: Client connection negotiation failed: client connection to 127.2.36.125:42825: connect: Connection refused (error 111)
I20250115 02:53:15.382701 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:42825
I20250115 02:53:15.382836 9804 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:42825 every 8 connection(s)
I20250115 02:53:15.386828 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:15.387039 9805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 02:53:15.393525 9807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:15.394475 9808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:15.395558 9810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:15.396158 9805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:15.396869 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:15.397799 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:15.398042 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:15.398234 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909595398218 us; error 0 us; skew 500 ppm
I20250115 02:53:15.398869 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:15.400323 9740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } attempt: 2
I20250115 02:53:15.402870 2193 webserver.cc:458] Webserver started at http://127.2.36.124:38139/ using document root <none> and password file <none>
I20250115 02:53:15.403527 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:15.403764 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:15.404069 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:15.405443 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "87d6ee7fe59a451c8beda99e8d21e9db"
format_stamp: "Formatted at 2025-01-15 02:53:15 on dist-test-slave-1lv2"
I20250115 02:53:15.408164 9805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:15.416267 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.009s sys 0.003s
I20250115 02:53:15.416628 9740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:15.422502 9740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:36699: Network error: Client connection negotiation failed: client connection to 127.2.36.124:36699: connect: Connection refused (error 111)
I20250115 02:53:15.427277 9818 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:15.427865 9805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:15.428283 2193 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.000s sys 0.005s
I20250115 02:53:15.428674 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "87d6ee7fe59a451c8beda99e8d21e9db"
format_stamp: "Formatted at 2025-01-15 02:53:15 on dist-test-slave-1lv2"
I20250115 02:53:15.429036 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250115 02:53:15.433549 9805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:36699: Network error: Client connection negotiation failed: client connection to 127.2.36.124:36699: connect: Connection refused (error 111)
I20250115 02:53:15.446408 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:15.447619 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:15.466050 9740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } attempt: 1
W20250115 02:53:15.470947 9740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:36699: Network error: Client connection negotiation failed: client connection to 127.2.36.124:36699: connect: Connection refused (error 111)
I20250115 02:53:15.484228 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:36699
I20250115 02:53:15.484325 9870 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:36699 every 8 connection(s)
I20250115 02:53:15.487232 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:53:15.487941 9871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:15.493970 9871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:15.496284 9805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } attempt: 1
I20250115 02:53:15.505213 9871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:15.514060 9805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4: Bootstrap starting.
I20250115 02:53:15.515091 9871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:15.519176 9805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:15.523579 9805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4: No bootstrap required, opened a new log
I20250115 02:53:15.525995 9805 raft_consensus.cc:357] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } }
I20250115 02:53:15.526723 9805 raft_consensus.cc:383] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:15.527032 9805 raft_consensus.cc:738] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7660636b6546f5bc9c08a052af62c4, State: Initialized, Role: FOLLOWER
I20250115 02:53:15.527711 9805 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } }
I20250115 02:53:15.529687 9876 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } } }
I20250115 02:53:15.530256 9876 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:15.531459 9805 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:15.531786 9871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db: Bootstrap starting.
I20250115 02:53:15.537983 9871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:15.543239 9871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db: No bootstrap required, opened a new log
I20250115 02:53:15.545399 9871 raft_consensus.cc:357] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } }
I20250115 02:53:15.545902 9871 raft_consensus.cc:383] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:15.546123 9871 raft_consensus.cc:738] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d6ee7fe59a451c8beda99e8d21e9db, State: Initialized, Role: FOLLOWER
I20250115 02:53:15.546649 9871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } }
W20250115 02:53:15.546988 9888 catalog_manager.cc:1559] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:15.547279 9888 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:15.548241 9889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } } }
I20250115 02:53:15.548816 9889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:15.549517 9871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:15.552511 9740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } attempt: 2
W20250115 02:53:15.565403 9900 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:15.565794 9900 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:15.568766 9740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b: Bootstrap starting.
I20250115 02:53:15.573035 9740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:15.576809 9740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b: No bootstrap required, opened a new log
I20250115 02:53:15.578703 9740 raft_consensus.cc:357] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } }
I20250115 02:53:15.579274 9740 raft_consensus.cc:383] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:15.579483 9740 raft_consensus.cc:738] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22aa6472c6b94725b1f14899990faf4b, State: Initialized, Role: FOLLOWER
I20250115 02:53:15.579952 9740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } }
I20250115 02:53:15.581449 9902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } } }
I20250115 02:53:15.582015 9902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:15.582676 9740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:15.592664 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:53:15.592979 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:53:15.594853 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:15.596843 9913 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:15.597215 9913 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:53:15.602569 9914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:15.603257 9915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:15.604295 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:15.604907 9917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:15.605722 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:15.605924 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:15.606066 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909595606051 us; error 0 us; skew 500 ppm
I20250115 02:53:15.606531 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:15.608887 2193 webserver.cc:458] Webserver started at http://127.2.36.65:42829/ using document root <none> and password file <none>
I20250115 02:53:15.609324 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:15.609477 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:15.609678 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:15.610659 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8"
format_stamp: "Formatted at 2025-01-15 02:53:15 on dist-test-slave-1lv2"
I20250115 02:53:15.614893 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250115 02:53:15.618188 9922 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:15.619058 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:15.619081 9876 raft_consensus.cc:491] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:15.619468 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8"
format_stamp: "Formatted at 2025-01-15 02:53:15 on dist-test-slave-1lv2"
I20250115 02:53:15.619642 9876 raft_consensus.cc:513] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } }
I20250115 02:53:15.619824 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:15.621258 9876 leader_election.cc:290] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22aa6472c6b94725b1f14899990faf4b (127.2.36.126:33631), 87d6ee7fe59a451c8beda99e8d21e9db (127.2.36.124:36699)
I20250115 02:53:15.622144 9715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc7660636b6546f5bc9c08a052af62c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22aa6472c6b94725b1f14899990faf4b" is_pre_election: true
I20250115 02:53:15.622311 9846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc7660636b6546f5bc9c08a052af62c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" is_pre_election: true
I20250115 02:53:15.622941 9715 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc7660636b6546f5bc9c08a052af62c4 in term 0.
I20250115 02:53:15.623121 9846 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc7660636b6546f5bc9c08a052af62c4 in term 0.
I20250115 02:53:15.624310 9757 leader_election.cc:304] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87d6ee7fe59a451c8beda99e8d21e9db, bc7660636b6546f5bc9c08a052af62c4; no voters:
I20250115 02:53:15.625074 9876 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:15.625419 9876 raft_consensus.cc:491] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:15.625674 9876 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:15.630748 9876 raft_consensus.cc:513] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } }
I20250115 02:53:15.632292 9876 leader_election.cc:290] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [CANDIDATE]: Term 1 election: Requested vote from peers 22aa6472c6b94725b1f14899990faf4b (127.2.36.126:33631), 87d6ee7fe59a451c8beda99e8d21e9db (127.2.36.124:36699)
I20250115 02:53:15.633173 9715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc7660636b6546f5bc9c08a052af62c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22aa6472c6b94725b1f14899990faf4b"
I20250115 02:53:15.633400 9846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc7660636b6546f5bc9c08a052af62c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d6ee7fe59a451c8beda99e8d21e9db"
I20250115 02:53:15.633721 9715 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:15.633980 9846 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:15.638926 9715 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc7660636b6546f5bc9c08a052af62c4 in term 1.
I20250115 02:53:15.639911 9756 leader_election.cc:304] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22aa6472c6b94725b1f14899990faf4b, bc7660636b6546f5bc9c08a052af62c4; no voters:
I20250115 02:53:15.640625 9876 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:15.640728 9846 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc7660636b6546f5bc9c08a052af62c4 in term 1.
I20250115 02:53:15.641803 9876 raft_consensus.cc:695] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 1 LEADER]: Becoming Leader. State: Replica: bc7660636b6546f5bc9c08a052af62c4, State: Running, Role: LEADER
I20250115 02:53:15.642596 9876 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } }
I20250115 02:53:15.647485 9927 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7660636b6546f5bc9c08a052af62c4. Latest consensus state: current_term: 1 leader_uuid: "bc7660636b6546f5bc9c08a052af62c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } } }
I20250115 02:53:15.648244 9927 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:15.649629 9929 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:15.654224 9929 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:15.655553 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:15.656636 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:15.658179 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:15.661181 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:15.661427 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:15.661696 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:15.661898 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:15.663057 9846 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [term 1 FOLLOWER]: Refusing update from remote peer bc7660636b6546f5bc9c08a052af62c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:15.664307 9715 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [term 1 FOLLOWER]: Refusing update from remote peer bc7660636b6546f5bc9c08a052af62c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:15.664346 9927 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 02:53:15.665598 9876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:15.675804 9902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7660636b6546f5bc9c08a052af62c4. Latest consensus state: current_term: 1 leader_uuid: "bc7660636b6546f5bc9c08a052af62c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } } }
I20250115 02:53:15.676659 9902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:15.676651 9889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7660636b6546f5bc9c08a052af62c4. Latest consensus state: current_term: 1 leader_uuid: "bc7660636b6546f5bc9c08a052af62c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } } }
I20250115 02:53:15.677565 9889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:15.683315 9876 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc7660636b6546f5bc9c08a052af62c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } } }
I20250115 02:53:15.684401 9876 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:15.685101 9876 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc7660636b6546f5bc9c08a052af62c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } } }
I20250115 02:53:15.685810 9876 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:15.685885 9929 catalog_manager.cc:1348] Generated new cluster ID: 8e08e4592a214e28a3460674412d592e
I20250115 02:53:15.686331 9929 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:15.702569 9902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc7660636b6546f5bc9c08a052af62c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } } }
I20250115 02:53:15.703413 9902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:15.708164 9889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc7660636b6546f5bc9c08a052af62c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa6472c6b94725b1f14899990faf4b" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33631 } } peers { permanent_uuid: "bc7660636b6546f5bc9c08a052af62c4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 42825 } } peers { permanent_uuid: "87d6ee7fe59a451c8beda99e8d21e9db" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36699 } } }
I20250115 02:53:15.709033 9889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:15.726866 9929 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:53:15.728614 9929 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:15.753655 9929 catalog_manager.cc:5899] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4: Generated new TSK 0
I20250115 02:53:15.754393 9929 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:15.770356 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:39557
I20250115 02:53:15.770449 9996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:39557 every 8 connection(s)
I20250115 02:53:15.781667 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:15.790865 9997 heartbeater.cc:346] Connected to a master server at 127.2.36.124:36699
I20250115 02:53:15.791361 9997 heartbeater.cc:463] Registering TS with master...
W20250115 02:53:15.791407 10004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:15.792449 9997 heartbeater.cc:509] Master 127.2.36.124:36699 requested a full tablet report, sending...
W20250115 02:53:15.792732 10005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:15.795711 9836 ts_manager.cc:194] Registered new tserver with Master: 2c5ad67fecee4dc1849b2fbcdb5b9ab8 (127.2.36.65:39557)
I20250115 02:53:15.799234 9998 heartbeater.cc:346] Connected to a master server at 127.2.36.125:42825
I20250115 02:53:15.799535 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:15.799687 9998 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:15.800590 9998 heartbeater.cc:509] Master 127.2.36.125:42825 requested a full tablet report, sending...
W20250115 02:53:15.802070 10007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:15.802780 9999 heartbeater.cc:346] Connected to a master server at 127.2.36.126:33631
I20250115 02:53:15.802801 9770 ts_manager.cc:194] Registered new tserver with Master: 2c5ad67fecee4dc1849b2fbcdb5b9ab8 (127.2.36.65:39557)
I20250115 02:53:15.803099 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250115 02:53:15.803359 9999 heartbeater.cc:463] Registering TS with master...
W20250115 02:53:15.803488 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:15.803788 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909595803760 us; error 0 us; skew 500 ppm
I20250115 02:53:15.804291 9999 heartbeater.cc:509] Master 127.2.36.126:33631 requested a full tablet report, sending...
I20250115 02:53:15.804572 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:15.804869 9770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34126
I20250115 02:53:15.806885 9705 ts_manager.cc:194] Registered new tserver with Master: 2c5ad67fecee4dc1849b2fbcdb5b9ab8 (127.2.36.65:39557)
I20250115 02:53:15.808799 2193 webserver.cc:458] Webserver started at http://127.2.36.66:44171/ using document root <none> and password file <none>
I20250115 02:53:15.809301 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:15.809481 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:15.809743 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:15.810859 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "22cf68f5b461447aa77a2b2522983773"
format_stamp: "Formatted at 2025-01-15 02:53:15 on dist-test-slave-1lv2"
I20250115 02:53:15.814954 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250115 02:53:15.817970 9998 heartbeater.cc:501] Master 127.2.36.125:42825 was elected leader, sending a full tablet report...
I20250115 02:53:15.818090 10012 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:15.819057 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:15.819356 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "22cf68f5b461447aa77a2b2522983773"
format_stamp: "Formatted at 2025-01-15 02:53:15 on dist-test-slave-1lv2"
I20250115 02:53:15.819728 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:15.850999 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:15.852113 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:15.853274 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:15.855568 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:15.855794 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:15.856016 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:15.856168 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:15.900090 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:43145
I20250115 02:53:15.900179 10074 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:43145 every 8 connection(s)
I20250115 02:53:15.922194 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:15.925345 10075 heartbeater.cc:346] Connected to a master server at 127.2.36.124:36699
I20250115 02:53:15.925904 10075 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:15.926774 10075 heartbeater.cc:509] Master 127.2.36.124:36699 requested a full tablet report, sending...
I20250115 02:53:15.927101 10076 heartbeater.cc:346] Connected to a master server at 127.2.36.125:42825
I20250115 02:53:15.927467 10076 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:15.928236 10076 heartbeater.cc:509] Master 127.2.36.125:42825 requested a full tablet report, sending...
I20250115 02:53:15.929574 9836 ts_manager.cc:194] Registered new tserver with Master: 22cf68f5b461447aa77a2b2522983773 (127.2.36.66:43145)
I20250115 02:53:15.931901 9770 ts_manager.cc:194] Registered new tserver with Master: 22cf68f5b461447aa77a2b2522983773 (127.2.36.66:43145)
I20250115 02:53:15.933797 9770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34132
W20250115 02:53:15.935011 10082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:15.938340 10078 heartbeater.cc:346] Connected to a master server at 127.2.36.126:33631
I20250115 02:53:15.938783 10078 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:15.939476 10078 heartbeater.cc:509] Master 127.2.36.126:33631 requested a full tablet report, sending...
W20250115 02:53:15.939791 10083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:15.942175 9705 ts_manager.cc:194] Registered new tserver with Master: 22cf68f5b461447aa77a2b2522983773 (127.2.36.66:43145)
W20250115 02:53:15.943354 10085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:15.944759 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:15.945516 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:15.945708 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:15.945854 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909595945844 us; error 0 us; skew 500 ppm
I20250115 02:53:15.946349 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:15.948443 10076 heartbeater.cc:501] Master 127.2.36.125:42825 was elected leader, sending a full tablet report...
I20250115 02:53:15.952268 2193 webserver.cc:458] Webserver started at http://127.2.36.67:41303/ using document root <none> and password file <none>
I20250115 02:53:15.952868 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:15.953047 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:15.953281 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:15.954483 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "c1f780775f37424c914963a73b477dee"
format_stamp: "Formatted at 2025-01-15 02:53:15 on dist-test-slave-1lv2"
I20250115 02:53:15.959321 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250115 02:53:15.964318 10090 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:15.965271 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250115 02:53:15.965658 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "c1f780775f37424c914963a73b477dee"
format_stamp: "Formatted at 2025-01-15 02:53:15 on dist-test-slave-1lv2"
I20250115 02:53:15.966044 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:15.988852 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:15.990290 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:15.992648 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:15.995527 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:15.995808 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:15.996063 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:15.996224 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:16.067008 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:41547
I20250115 02:53:16.067101 10152 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:41547 every 8 connection(s)
I20250115 02:53:16.103756 10154 heartbeater.cc:346] Connected to a master server at 127.2.36.125:42825
I20250115 02:53:16.104224 10154 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:16.104717 10153 heartbeater.cc:346] Connected to a master server at 127.2.36.124:36699
I20250115 02:53:16.105008 10154 heartbeater.cc:509] Master 127.2.36.125:42825 requested a full tablet report, sending...
I20250115 02:53:16.105137 10153 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:16.106086 10153 heartbeater.cc:509] Master 127.2.36.124:36699 requested a full tablet report, sending...
I20250115 02:53:16.107190 9770 ts_manager.cc:194] Registered new tserver with Master: c1f780775f37424c914963a73b477dee (127.2.36.67:41547)
I20250115 02:53:16.108973 9770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34142
I20250115 02:53:16.109119 9836 ts_manager.cc:194] Registered new tserver with Master: c1f780775f37424c914963a73b477dee (127.2.36.67:41547)
I20250115 02:53:16.113366 10155 heartbeater.cc:346] Connected to a master server at 127.2.36.126:33631
I20250115 02:53:16.113785 10155 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:16.114441 10155 heartbeater.cc:509] Master 127.2.36.126:33631 requested a full tablet report, sending...
I20250115 02:53:16.117851 9704 ts_manager.cc:194] Registered new tserver with Master: c1f780775f37424c914963a73b477dee (127.2.36.67:41547)
I20250115 02:53:16.117997 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.03256864s
I20250115 02:53:16.125033 10154 heartbeater.cc:501] Master 127.2.36.125:42825 was elected leader, sending a full tablet report...
I20250115 02:53:16.172935 9769 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34154:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:53:16.175292 9769 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:53:16.298292 10040 tablet_service.cc:1467] Processing CreateTablet for tablet 86b6cc040b8b4735b243a635c20186d2 (DEFAULT_TABLE table=ksck-test-table [id=e12472ab16f5452294c3aef3028c6619]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:16.300359 10040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86b6cc040b8b4735b243a635c20186d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:16.301214 10117 tablet_service.cc:1467] Processing CreateTablet for tablet c733ed7cafe04b908e029a71ef4dcb32 (DEFAULT_TABLE table=ksck-test-table [id=e12472ab16f5452294c3aef3028c6619]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:16.302595 10117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c733ed7cafe04b908e029a71ef4dcb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:16.302520 10116 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac7506e5870498cb7e6160f97cc5a1a (DEFAULT_TABLE table=ksck-test-table [id=e12472ab16f5452294c3aef3028c6619]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:16.303041 10118 tablet_service.cc:1467] Processing CreateTablet for tablet 86b6cc040b8b4735b243a635c20186d2 (DEFAULT_TABLE table=ksck-test-table [id=e12472ab16f5452294c3aef3028c6619]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:16.303843 10116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ac7506e5870498cb7e6160f97cc5a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:16.305999 10118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86b6cc040b8b4735b243a635c20186d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:16.321064 10038 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac7506e5870498cb7e6160f97cc5a1a (DEFAULT_TABLE table=ksck-test-table [id=e12472ab16f5452294c3aef3028c6619]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:16.321388 10039 tablet_service.cc:1467] Processing CreateTablet for tablet c733ed7cafe04b908e029a71ef4dcb32 (DEFAULT_TABLE table=ksck-test-table [id=e12472ab16f5452294c3aef3028c6619]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:16.320914 9962 tablet_service.cc:1467] Processing CreateTablet for tablet 86b6cc040b8b4735b243a635c20186d2 (DEFAULT_TABLE table=ksck-test-table [id=e12472ab16f5452294c3aef3028c6619]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:16.323271 9962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86b6cc040b8b4735b243a635c20186d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:16.322428 10038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ac7506e5870498cb7e6160f97cc5a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:16.329463 10039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c733ed7cafe04b908e029a71ef4dcb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:16.340335 9961 tablet_service.cc:1467] Processing CreateTablet for tablet c733ed7cafe04b908e029a71ef4dcb32 (DEFAULT_TABLE table=ksck-test-table [id=e12472ab16f5452294c3aef3028c6619]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:16.341708 9961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c733ed7cafe04b908e029a71ef4dcb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:16.343634 9960 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac7506e5870498cb7e6160f97cc5a1a (DEFAULT_TABLE table=ksck-test-table [id=e12472ab16f5452294c3aef3028c6619]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:16.344915 9960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ac7506e5870498cb7e6160f97cc5a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:16.379045 10174 tablet_bootstrap.cc:492] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee: Bootstrap starting.
I20250115 02:53:16.386168 10175 tablet_bootstrap.cc:492] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: Bootstrap starting.
I20250115 02:53:16.381615 10173 tablet_bootstrap.cc:492] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773: Bootstrap starting.
I20250115 02:53:16.395408 10174 tablet_bootstrap.cc:654] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:16.405129 10173 tablet_bootstrap.cc:654] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:16.412122 10174 tablet_bootstrap.cc:492] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee: No bootstrap required, opened a new log
I20250115 02:53:16.412695 10174 ts_tablet_manager.cc:1397] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.002s
I20250115 02:53:16.414289 10175 tablet_bootstrap.cc:654] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:16.415486 10174 raft_consensus.cc:357] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.416332 10174 raft_consensus.cc:383] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:16.416659 10174 raft_consensus.cc:738] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f780775f37424c914963a73b477dee, State: Initialized, Role: FOLLOWER
I20250115 02:53:16.417410 10173 tablet_bootstrap.cc:492] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773: No bootstrap required, opened a new log
I20250115 02:53:16.420902 10173 ts_tablet_manager.cc:1397] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.000s
I20250115 02:53:16.421358 10174 consensus_queue.cc:260] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.423669 10173 raft_consensus.cc:357] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } }
I20250115 02:53:16.428194 10173 raft_consensus.cc:383] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:16.428661 10173 raft_consensus.cc:738] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22cf68f5b461447aa77a2b2522983773, State: Initialized, Role: FOLLOWER
I20250115 02:53:16.430095 10173 consensus_queue.cc:260] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } }
I20250115 02:53:16.434837 10173 ts_tablet_manager.cc:1428] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250115 02:53:16.442270 10175 tablet_bootstrap.cc:492] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: No bootstrap required, opened a new log
I20250115 02:53:16.442888 10175 ts_tablet_manager.cc:1397] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.008s
I20250115 02:53:16.443001 10173 tablet_bootstrap.cc:492] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773: Bootstrap starting.
I20250115 02:53:16.449432 10173 tablet_bootstrap.cc:654] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:16.457028 10174 ts_tablet_manager.cc:1428] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee: Time spent starting tablet: real 0.044s user 0.010s sys 0.007s
I20250115 02:53:16.460503 10175 raft_consensus.cc:357] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } }
I20250115 02:53:16.461338 10175 raft_consensus.cc:383] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:16.461647 10175 raft_consensus.cc:738] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5ad67fecee4dc1849b2fbcdb5b9ab8, State: Initialized, Role: FOLLOWER
I20250115 02:53:16.462481 10175 consensus_queue.cc:260] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } }
I20250115 02:53:16.465294 10174 tablet_bootstrap.cc:492] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee: Bootstrap starting.
I20250115 02:53:16.471706 10174 tablet_bootstrap.cc:654] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:16.486311 10174 tablet_bootstrap.cc:492] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee: No bootstrap required, opened a new log
I20250115 02:53:16.486948 10174 ts_tablet_manager.cc:1397] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.004s
I20250115 02:53:16.493266 10174 raft_consensus.cc:357] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.494202 10174 raft_consensus.cc:383] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:16.494561 10174 raft_consensus.cc:738] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f780775f37424c914963a73b477dee, State: Initialized, Role: FOLLOWER
I20250115 02:53:16.495407 10174 consensus_queue.cc:260] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.495612 10173 tablet_bootstrap.cc:492] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773: No bootstrap required, opened a new log
I20250115 02:53:16.501118 10173 ts_tablet_manager.cc:1397] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773: Time spent bootstrapping tablet: real 0.058s user 0.017s sys 0.000s
I20250115 02:53:16.501788 10174 ts_tablet_manager.cc:1428] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250115 02:53:16.508626 10174 tablet_bootstrap.cc:492] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee: Bootstrap starting.
I20250115 02:53:16.515161 10174 tablet_bootstrap.cc:654] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:16.521009 10179 raft_consensus.cc:491] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:16.521634 10179 raft_consensus.cc:513] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } }
I20250115 02:53:16.524109 10179 leader_election.cc:290] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1f780775f37424c914963a73b477dee (127.2.36.67:41547), 2c5ad67fecee4dc1849b2fbcdb5b9ab8 (127.2.36.65:39557)
I20250115 02:53:16.508549 10173 raft_consensus.cc:357] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.535652 10173 raft_consensus.cc:383] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:16.536022 10173 raft_consensus.cc:738] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22cf68f5b461447aa77a2b2522983773, State: Initialized, Role: FOLLOWER
I20250115 02:53:16.540858 10175 ts_tablet_manager.cc:1428] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: Time spent starting tablet: real 0.098s user 0.015s sys 0.017s
I20250115 02:53:16.536790 10173 consensus_queue.cc:260] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.546331 10173 ts_tablet_manager.cc:1428] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773: Time spent starting tablet: real 0.045s user 0.001s sys 0.004s
I20250115 02:53:16.555979 10175 tablet_bootstrap.cc:492] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: Bootstrap starting.
I20250115 02:53:16.572940 9900 catalog_manager.cc:1260] Loaded cluster ID: 8e08e4592a214e28a3460674412d592e
I20250115 02:53:16.573403 9900 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db: loading cluster ID for follower catalog manager: success
I20250115 02:53:16.574349 10175 tablet_bootstrap.cc:654] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:16.580706 9900 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db: acquiring CA information for follower catalog manager: success
I20250115 02:53:16.583341 10173 tablet_bootstrap.cc:492] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773: Bootstrap starting.
I20250115 02:53:16.586194 10174 tablet_bootstrap.cc:492] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee: No bootstrap required, opened a new log
I20250115 02:53:16.601320 10128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b6cc040b8b4735b243a635c20186d2" candidate_uuid: "22cf68f5b461447aa77a2b2522983773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f780775f37424c914963a73b477dee" is_pre_election: true
I20250115 02:53:16.601794 10174 ts_tablet_manager.cc:1397] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee: Time spent bootstrapping tablet: real 0.093s user 0.035s sys 0.020s
I20250115 02:53:16.592804 9972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b6cc040b8b4735b243a635c20186d2" candidate_uuid: "22cf68f5b461447aa77a2b2522983773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" is_pre_election: true
I20250115 02:53:16.603096 9972 raft_consensus.cc:2463] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22cf68f5b461447aa77a2b2522983773 in term 0.
W20250115 02:53:16.603830 10016 leader_election.cc:343] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c1f780775f37424c914963a73b477dee (127.2.36.67:41547): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:16.604369 10180 raft_consensus.cc:491] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:16.604894 10180 raft_consensus.cc:513] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.605894 10173 tablet_bootstrap.cc:654] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:16.604616 10016 leader_election.cc:304] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22cf68f5b461447aa77a2b2522983773, 2c5ad67fecee4dc1849b2fbcdb5b9ab8; no voters: c1f780775f37424c914963a73b477dee
I20250115 02:53:16.607378 10179 raft_consensus.cc:2798] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:16.607767 10179 raft_consensus.cc:491] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:16.608093 10179 raft_consensus.cc:3054] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:16.611564 9913 catalog_manager.cc:1260] Loaded cluster ID: 8e08e4592a214e28a3460674412d592e
I20250115 02:53:16.611894 9913 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b: loading cluster ID for follower catalog manager: success
I20250115 02:53:16.607347 10180 leader_election.cc:290] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c5ad67fecee4dc1849b2fbcdb5b9ab8 (127.2.36.65:39557), 22cf68f5b461447aa77a2b2522983773 (127.2.36.66:43145)
I20250115 02:53:16.604578 10174 raft_consensus.cc:357] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } }
I20250115 02:53:16.618018 10174 raft_consensus.cc:383] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:16.618441 10174 raft_consensus.cc:738] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f780775f37424c914963a73b477dee, State: Initialized, Role: FOLLOWER
I20250115 02:53:16.619277 10174 consensus_queue.cc:260] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } }
I20250115 02:53:16.613246 9900 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:16.622460 10174 ts_tablet_manager.cc:1428] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20250115 02:53:16.629004 10179 raft_consensus.cc:513] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } }
I20250115 02:53:16.635298 10050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7506e5870498cb7e6160f97cc5a1a" candidate_uuid: "c1f780775f37424c914963a73b477dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22cf68f5b461447aa77a2b2522983773" is_pre_election: true
I20250115 02:53:16.640856 10179 leader_election.cc:290] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [CANDIDATE]: Term 1 election: Requested vote from peers c1f780775f37424c914963a73b477dee (127.2.36.67:41547), 2c5ad67fecee4dc1849b2fbcdb5b9ab8 (127.2.36.65:39557)
W20250115 02:53:16.648836 10093 leader_election.cc:343] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22cf68f5b461447aa77a2b2522983773 (127.2.36.66:43145): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:16.651082 10128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b6cc040b8b4735b243a635c20186d2" candidate_uuid: "22cf68f5b461447aa77a2b2522983773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f780775f37424c914963a73b477dee"
I20250115 02:53:16.651607 10128 raft_consensus.cc:3054] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:16.656000 9913 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b: acquiring CA information for follower catalog manager: success
I20250115 02:53:16.659233 10173 tablet_bootstrap.cc:492] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773: No bootstrap required, opened a new log
I20250115 02:53:16.659873 10173 ts_tablet_manager.cc:1397] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773: Time spent bootstrapping tablet: real 0.077s user 0.021s sys 0.013s
I20250115 02:53:16.662609 9913 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:16.662820 10173 raft_consensus.cc:357] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.663661 10173 raft_consensus.cc:383] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:16.663982 10173 raft_consensus.cc:738] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22cf68f5b461447aa77a2b2522983773, State: Initialized, Role: FOLLOWER
I20250115 02:53:16.664718 10173 consensus_queue.cc:260] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.666249 9972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b6cc040b8b4735b243a635c20186d2" candidate_uuid: "22cf68f5b461447aa77a2b2522983773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8"
I20250115 02:53:16.666913 9972 raft_consensus.cc:3054] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:16.670158 10173 ts_tablet_manager.cc:1428] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250115 02:53:16.671734 10128 raft_consensus.cc:2463] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22cf68f5b461447aa77a2b2522983773 in term 1.
I20250115 02:53:16.679028 10016 leader_election.cc:304] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22cf68f5b461447aa77a2b2522983773, c1f780775f37424c914963a73b477dee; no voters:
I20250115 02:53:16.682824 9972 raft_consensus.cc:2463] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22cf68f5b461447aa77a2b2522983773 in term 1.
I20250115 02:53:16.683998 10179 raft_consensus.cc:2798] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:16.687937 9971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7506e5870498cb7e6160f97cc5a1a" candidate_uuid: "c1f780775f37424c914963a73b477dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" is_pre_election: true
W20250115 02:53:16.690357 10094 leader_election.cc:343] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2c5ad67fecee4dc1849b2fbcdb5b9ab8 (127.2.36.65:39557): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:16.691457 10094 leader_election.cc:304] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1f780775f37424c914963a73b477dee; no voters: 22cf68f5b461447aa77a2b2522983773, 2c5ad67fecee4dc1849b2fbcdb5b9ab8
I20250115 02:53:16.692401 10180 raft_consensus.cc:2743] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:53:16.696282 10175 tablet_bootstrap.cc:492] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: No bootstrap required, opened a new log
I20250115 02:53:16.697810 10179 raft_consensus.cc:695] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 1 LEADER]: Becoming Leader. State: Replica: 22cf68f5b461447aa77a2b2522983773, State: Running, Role: LEADER
I20250115 02:53:16.696864 10175 ts_tablet_manager.cc:1397] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: Time spent bootstrapping tablet: real 0.141s user 0.012s sys 0.004s
I20250115 02:53:16.698601 10179 consensus_queue.cc:237] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } }
I20250115 02:53:16.700776 10175 raft_consensus.cc:357] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.701483 10175 raft_consensus.cc:383] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:16.702075 10175 raft_consensus.cc:738] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5ad67fecee4dc1849b2fbcdb5b9ab8, State: Initialized, Role: FOLLOWER
I20250115 02:53:16.702903 10175 consensus_queue.cc:260] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.704939 10175 ts_tablet_manager.cc:1428] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250115 02:53:16.705982 10175 tablet_bootstrap.cc:492] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: Bootstrap starting.
I20250115 02:53:16.712730 10175 tablet_bootstrap.cc:654] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:16.719727 10175 tablet_bootstrap.cc:492] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: No bootstrap required, opened a new log
I20250115 02:53:16.720319 10175 ts_tablet_manager.cc:1397] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20250115 02:53:16.723191 10175 raft_consensus.cc:357] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.724009 10175 raft_consensus.cc:383] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:16.724347 10175 raft_consensus.cc:738] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5ad67fecee4dc1849b2fbcdb5b9ab8, State: Initialized, Role: FOLLOWER
I20250115 02:53:16.725198 10175 consensus_queue.cc:260] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.725415 9768 catalog_manager.cc:5526] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22cf68f5b461447aa77a2b2522983773 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "22cf68f5b461447aa77a2b2522983773" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:16.732623 10175 ts_tablet_manager.cc:1428] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20250115 02:53:16.787701 10180 raft_consensus.cc:491] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:16.788136 10180 raft_consensus.cc:513] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.789952 10180 leader_election.cc:290] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c5ad67fecee4dc1849b2fbcdb5b9ab8 (127.2.36.65:39557), 22cf68f5b461447aa77a2b2522983773 (127.2.36.66:43145)
I20250115 02:53:16.790889 10050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c733ed7cafe04b908e029a71ef4dcb32" candidate_uuid: "c1f780775f37424c914963a73b477dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22cf68f5b461447aa77a2b2522983773" is_pre_election: true
I20250115 02:53:16.791591 10050 raft_consensus.cc:2463] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1f780775f37424c914963a73b477dee in term 0.
I20250115 02:53:16.790892 9972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c733ed7cafe04b908e029a71ef4dcb32" candidate_uuid: "c1f780775f37424c914963a73b477dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" is_pre_election: true
I20250115 02:53:16.792905 9972 raft_consensus.cc:2463] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1f780775f37424c914963a73b477dee in term 0.
I20250115 02:53:16.793843 10094 leader_election.cc:304] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c5ad67fecee4dc1849b2fbcdb5b9ab8, c1f780775f37424c914963a73b477dee; no voters:
I20250115 02:53:16.795848 10180 raft_consensus.cc:2798] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:16.796180 10180 raft_consensus.cc:491] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:16.796492 10180 raft_consensus.cc:3054] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:16.802548 10180 raft_consensus.cc:513] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.805640 10050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c733ed7cafe04b908e029a71ef4dcb32" candidate_uuid: "c1f780775f37424c914963a73b477dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22cf68f5b461447aa77a2b2522983773"
I20250115 02:53:16.806011 9972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c733ed7cafe04b908e029a71ef4dcb32" candidate_uuid: "c1f780775f37424c914963a73b477dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8"
I20250115 02:53:16.806319 10050 raft_consensus.cc:3054] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:16.806674 9972 raft_consensus.cc:3054] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:16.807430 10180 leader_election.cc:290] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [CANDIDATE]: Term 1 election: Requested vote from peers 2c5ad67fecee4dc1849b2fbcdb5b9ab8 (127.2.36.65:39557), 22cf68f5b461447aa77a2b2522983773 (127.2.36.66:43145)
I20250115 02:53:16.810205 10179 raft_consensus.cc:491] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:16.810735 10179 raft_consensus.cc:513] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.812736 10179 leader_election.cc:290] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c5ad67fecee4dc1849b2fbcdb5b9ab8 (127.2.36.65:39557), c1f780775f37424c914963a73b477dee (127.2.36.67:41547)
I20250115 02:53:16.814613 9971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7506e5870498cb7e6160f97cc5a1a" candidate_uuid: "22cf68f5b461447aa77a2b2522983773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" is_pre_election: true
I20250115 02:53:16.814868 10128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7506e5870498cb7e6160f97cc5a1a" candidate_uuid: "22cf68f5b461447aa77a2b2522983773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f780775f37424c914963a73b477dee" is_pre_election: true
I20250115 02:53:16.815338 9971 raft_consensus.cc:2463] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22cf68f5b461447aa77a2b2522983773 in term 0.
I20250115 02:53:16.815479 10128 raft_consensus.cc:2463] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22cf68f5b461447aa77a2b2522983773 in term 0.
I20250115 02:53:16.816649 10016 leader_election.cc:304] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22cf68f5b461447aa77a2b2522983773, c1f780775f37424c914963a73b477dee; no voters:
I20250115 02:53:16.818429 10179 raft_consensus.cc:2798] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:16.818841 10179 raft_consensus.cc:491] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:16.819162 10179 raft_consensus.cc:3054] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:16.825567 10050 raft_consensus.cc:2463] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1f780775f37424c914963a73b477dee in term 1.
I20250115 02:53:16.813719 9972 raft_consensus.cc:2463] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1f780775f37424c914963a73b477dee in term 1.
I20250115 02:53:16.825485 10179 raft_consensus.cc:513] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.826534 10093 leader_election.cc:304] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22cf68f5b461447aa77a2b2522983773, c1f780775f37424c914963a73b477dee; no voters:
I20250115 02:53:16.828078 10179 leader_election.cc:290] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [CANDIDATE]: Term 1 election: Requested vote from peers 2c5ad67fecee4dc1849b2fbcdb5b9ab8 (127.2.36.65:39557), c1f780775f37424c914963a73b477dee (127.2.36.67:41547)
I20250115 02:53:16.829455 10180 raft_consensus.cc:2798] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:16.831203 10128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7506e5870498cb7e6160f97cc5a1a" candidate_uuid: "22cf68f5b461447aa77a2b2522983773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f780775f37424c914963a73b477dee"
I20250115 02:53:16.831838 10128 raft_consensus.cc:3054] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:16.841650 10128 raft_consensus.cc:2463] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22cf68f5b461447aa77a2b2522983773 in term 1.
I20250115 02:53:16.842705 10016 leader_election.cc:304] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22cf68f5b461447aa77a2b2522983773, c1f780775f37424c914963a73b477dee; no voters:
I20250115 02:53:16.843191 10180 raft_consensus.cc:695] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 1 LEADER]: Becoming Leader. State: Replica: c1f780775f37424c914963a73b477dee, State: Running, Role: LEADER
I20250115 02:53:16.842624 9972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7506e5870498cb7e6160f97cc5a1a" candidate_uuid: "22cf68f5b461447aa77a2b2522983773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8"
I20250115 02:53:16.844769 9972 raft_consensus.cc:3054] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:16.848809 10179 raft_consensus.cc:2798] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:16.848714 10180 consensus_queue.cc:237] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.850790 10179 raft_consensus.cc:695] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 1 LEADER]: Becoming Leader. State: Replica: 22cf68f5b461447aa77a2b2522983773, State: Running, Role: LEADER
I20250115 02:53:16.851657 10179 consensus_queue.cc:237] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } }
I20250115 02:53:16.858325 9972 raft_consensus.cc:2463] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22cf68f5b461447aa77a2b2522983773 in term 1.
I20250115 02:53:16.871701 9769 catalog_manager.cc:5526] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee reported cstate change: term changed from 0 to 1, leader changed from <none> to c1f780775f37424c914963a73b477dee (127.2.36.67). New cstate: current_term: 1 leader_uuid: "c1f780775f37424c914963a73b477dee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } health_report { overall_health: HEALTHY } } }
I20250115 02:53:16.872193 9768 catalog_manager.cc:5526] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22cf68f5b461447aa77a2b2522983773 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "22cf68f5b461447aa77a2b2522983773" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:17.154122 10179 consensus_queue.cc:1035] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:17.218444 10179 consensus_queue.cc:1035] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:17.310745 10180 consensus_queue.cc:1035] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [LEADER]: Connected to new peer: Peer: permanent_uuid: "22cf68f5b461447aa77a2b2522983773" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:17.334123 10180 consensus_queue.cc:1035] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 02:53:17.368846 10179 consensus_queue.cc:1035] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1f780775f37424c914963a73b477dee" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 41547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:17.392745 10179 consensus_queue.cc:1035] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c5ad67fecee4dc1849b2fbcdb5b9ab8" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:17.662379 10040 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250115 02:53:17.698434 9961 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:53:17.762660 10117 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:53:18.081068 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:53:18.146397 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:18.147003 2193 tablet_replica.cc:331] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: stopping tablet replica
I20250115 02:53:18.147553 2193 raft_consensus.cc:2238] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:18.147969 2193 raft_consensus.cc:2267] T c733ed7cafe04b908e029a71ef4dcb32 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:18.149993 2193 tablet_replica.cc:331] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: stopping tablet replica
I20250115 02:53:18.150599 2193 raft_consensus.cc:2238] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:18.151114 2193 raft_consensus.cc:2267] T 5ac7506e5870498cb7e6160f97cc5a1a P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:18.156770 2193 tablet_replica.cc:331] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8: stopping tablet replica
I20250115 02:53:18.157502 2193 raft_consensus.cc:2238] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:18.157971 2193 raft_consensus.cc:2267] T 86b6cc040b8b4735b243a635c20186d2 P 2c5ad67fecee4dc1849b2fbcdb5b9ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:18.179808 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:53:18.194351 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:53:18.227695 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:18.228302 2193 tablet_replica.cc:331] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773: stopping tablet replica
I20250115 02:53:18.229074 2193 raft_consensus.cc:2238] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:18.229890 2193 raft_consensus.cc:2267] T 5ac7506e5870498cb7e6160f97cc5a1a P 22cf68f5b461447aa77a2b2522983773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:18.232147 2193 tablet_replica.cc:331] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773: stopping tablet replica
I20250115 02:53:18.232692 2193 raft_consensus.cc:2238] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:18.233153 2193 raft_consensus.cc:2267] T c733ed7cafe04b908e029a71ef4dcb32 P 22cf68f5b461447aa77a2b2522983773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:18.235355 2193 tablet_replica.cc:331] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773: stopping tablet replica
I20250115 02:53:18.235939 2193 raft_consensus.cc:2238] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:18.236645 2193 raft_consensus.cc:2267] T 86b6cc040b8b4735b243a635c20186d2 P 22cf68f5b461447aa77a2b2522983773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:18.259675 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:53:18.273993 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:53:18.295997 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:18.296519 2193 tablet_replica.cc:331] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee: stopping tablet replica
I20250115 02:53:18.297039 2193 raft_consensus.cc:2238] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:18.297408 2193 raft_consensus.cc:2267] T 86b6cc040b8b4735b243a635c20186d2 P c1f780775f37424c914963a73b477dee [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:18.299221 2193 tablet_replica.cc:331] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee: stopping tablet replica
I20250115 02:53:18.299697 2193 raft_consensus.cc:2238] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:18.300042 2193 raft_consensus.cc:2267] T 5ac7506e5870498cb7e6160f97cc5a1a P c1f780775f37424c914963a73b477dee [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:18.301942 2193 tablet_replica.cc:331] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee: stopping tablet replica
I20250115 02:53:18.302359 2193 raft_consensus.cc:2238] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:18.303071 2193 raft_consensus.cc:2267] T c733ed7cafe04b908e029a71ef4dcb32 P c1f780775f37424c914963a73b477dee [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:18.323145 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:53:18.336891 2193 master.cc:537] Master@127.2.36.126:33631 shutting down...
I20250115 02:53:18.352972 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:18.353500 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:18.353806 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 22aa6472c6b94725b1f14899990faf4b: stopping tablet replica
I20250115 02:53:18.372869 2193 master.cc:559] Master@127.2.36.126:33631 shutdown complete.
I20250115 02:53:18.384043 2193 master.cc:537] Master@127.2.36.125:42825 shutting down...
W20250115 02:53:18.399457 9756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.36.126:33631: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20250115 02:53:18.401198 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:18.402076 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:18.402393 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P bc7660636b6546f5bc9c08a052af62c4: stopping tablet replica
I20250115 02:53:18.422135 2193 master.cc:559] Master@127.2.36.125:42825 shutdown complete.
I20250115 02:53:18.434106 2193 master.cc:537] Master@127.2.36.124:36699 shutting down...
I20250115 02:53:18.448580 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:18.449083 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:18.449359 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 87d6ee7fe59a451c8beda99e8d21e9db: stopping tablet replica
I20250115 02:53:18.467432 2193 master.cc:559] Master@127.2.36.124:36699 shutdown complete.
[ OK ] RemoteKsckTest.TestNonMatchingTableFilter (3277 ms)
[ RUN ] RemoteKsckTest.TestMatchingTableFilter
I20250115 02:53:18.495803 2193 test_util.cc:274] Using random seed: -945810394
I20250115 02:53:18.496738 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:40125,127.2.36.125:40819,127.2.36.124:45789
I20250115 02:53:18.497958 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:18.503443 10247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:18.504079 10248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:18.505151 10250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:18.505833 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:18.506568 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:18.506793 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:18.506942 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909598506925 us; error 0 us; skew 500 ppm
I20250115 02:53:18.507422 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:18.509586 2193 webserver.cc:458] Webserver started at http://127.2.36.126:37893/ using document root <none> and password file <none>
I20250115 02:53:18.510032 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:18.510202 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:18.510435 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:18.511498 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "9be9a98525244abfa67175d705720235"
format_stamp: "Formatted at 2025-01-15 02:53:18 on dist-test-slave-1lv2"
I20250115 02:53:18.515698 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20250115 02:53:18.518847 10255 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:18.519587 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:53:18.519860 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "9be9a98525244abfa67175d705720235"
format_stamp: "Formatted at 2025-01-15 02:53:18 on dist-test-slave-1lv2"
I20250115 02:53:18.520115 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:18.553223 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:18.554167 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:18.586444 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:40125
I20250115 02:53:18.586532 10306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:40125 every 8 connection(s)
I20250115 02:53:18.589900 10307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:18.590091 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:18.595997 10309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:18.596599 10310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:18.597177 10307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:18.599457 10312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:18.601336 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:18.602079 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:18.602268 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:18.602417 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909598602402 us; error 0 us; skew 500 ppm
I20250115 02:53:18.602952 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:18.605595 2193 webserver.cc:458] Webserver started at http://127.2.36.125:45459/ using document root <none> and password file <none>
I20250115 02:53:18.606115 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:18.606292 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:18.606521 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:18.607805 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c"
format_stamp: "Formatted at 2025-01-15 02:53:18 on dist-test-slave-1lv2"
I20250115 02:53:18.608860 10307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:18.612550 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
W20250115 02:53:18.613634 10307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:40819: Network error: Client connection negotiation failed: client connection to 127.2.36.125:40819: connect: Connection refused (error 111)
I20250115 02:53:18.615849 10320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:18.616582 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:53:18.616847 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c"
format_stamp: "Formatted at 2025-01-15 02:53:18 on dist-test-slave-1lv2"
I20250115 02:53:18.617090 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:18.632848 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:18.633832 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:18.664145 10307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } attempt: 1
W20250115 02:53:18.669754 10307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:40819: Network error: Client connection negotiation failed: client connection to 127.2.36.125:40819: connect: Connection refused (error 111)
I20250115 02:53:18.672276 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:40819
I20250115 02:53:18.672362 10371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:40819 every 8 connection(s)
I20250115 02:53:18.675872 10372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:18.676029 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:18.681824 10374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:18.682484 10375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:18.683000 10372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:18.686002 10377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:18.686628 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:18.687614 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:18.687875 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:18.688078 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909598688061 us; error 0 us; skew 500 ppm
I20250115 02:53:18.688664 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:18.691435 2193 webserver.cc:458] Webserver started at http://127.2.36.124:35573/ using document root <none> and password file <none>
I20250115 02:53:18.691998 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:18.692224 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:18.692493 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:18.693581 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "7aeaf33e08c04784847da73e162f05af"
format_stamp: "Formatted at 2025-01-15 02:53:18 on dist-test-slave-1lv2"
I20250115 02:53:18.694690 10372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:18.698539 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20250115 02:53:18.702544 10384 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:18.703464 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250115 02:53:18.703816 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "7aeaf33e08c04784847da73e162f05af"
format_stamp: "Formatted at 2025-01-15 02:53:18 on dist-test-slave-1lv2"
I20250115 02:53:18.704197 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:18.705346 10372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:18.710476 10372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45789: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45789: connect: Connection refused (error 111)
I20250115 02:53:18.714157 10307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } attempt: 2
I20250115 02:53:18.722714 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:18.723218 10307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:18.723862 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250115 02:53:18.727499 10307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45789: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45789: connect: Connection refused (error 111)
I20250115 02:53:18.757931 10372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } attempt: 1
I20250115 02:53:18.759101 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:45789
I20250115 02:53:18.759176 10436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:45789 every 8 connection(s)
I20250115 02:53:18.761057 10307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } attempt: 1
I20250115 02:53:18.764138 10438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:18.773051 10438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:18.782433 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:53:18.783459 10372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c: Bootstrap starting.
I20250115 02:53:18.789309 10372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:18.789979 10307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235: Bootstrap starting.
I20250115 02:53:18.790834 10438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:18.795538 10307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:18.796191 10372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c: No bootstrap required, opened a new log
I20250115 02:53:18.798883 10372 raft_consensus.cc:357] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:18.799705 10372 raft_consensus.cc:383] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:18.799926 10372 raft_consensus.cc:738] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31dc9f4ea6ab494e8a4c3dddb7776b9c, State: Initialized, Role: FOLLOWER
I20250115 02:53:18.800632 10372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:18.802168 10307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235: No bootstrap required, opened a new log
I20250115 02:53:18.802915 10438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:18.804780 10307 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:18.805538 10307 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:18.805856 10307 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be9a98525244abfa67175d705720235, State: Initialized, Role: FOLLOWER
I20250115 02:53:18.806576 10307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:18.808566 10444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:18.809230 10444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:18.811700 10372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:18.812503 10445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:18.813350 10445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:18.814148 10307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:18.824193 10438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af: Bootstrap starting.
I20250115 02:53:18.829983 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:53:18.830415 10438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af: Neither blocks nor log segments found. Creating new log.
W20250115 02:53:18.834548 10464 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:18.835000 10464 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:18.837209 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:53:18.839802 10438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af: No bootstrap required, opened a new log
W20250115 02:53:18.841286 10468 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:18.841624 10468 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:18.842437 10438 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:18.843021 10438 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:18.843214 10438 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aeaf33e08c04784847da73e162f05af, State: Initialized, Role: FOLLOWER
I20250115 02:53:18.843717 10438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:18.845538 10469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:18.846109 10469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:18.846812 10438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:18.857352 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:18.859160 10480 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:18.859454 10480 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:53:18.863701 10481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:18.864619 10482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:18.865556 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:18.866114 10484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:18.866899 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:18.867107 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:18.867261 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909598867247 us; error 0 us; skew 500 ppm
I20250115 02:53:18.867774 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:18.869994 2193 webserver.cc:458] Webserver started at http://127.2.36.65:46269/ using document root <none> and password file <none>
I20250115 02:53:18.870429 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:18.870599 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:18.870872 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:18.871889 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "fcb481af490f4be18851faed3b37b812"
format_stamp: "Formatted at 2025-01-15 02:53:18 on dist-test-slave-1lv2"
I20250115 02:53:18.875926 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:53:18.878949 10489 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:18.879652 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:18.879936 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "fcb481af490f4be18851faed3b37b812"
format_stamp: "Formatted at 2025-01-15 02:53:18 on dist-test-slave-1lv2"
I20250115 02:53:18.880198 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:18.895107 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:18.896072 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:18.897368 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:18.899500 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:18.899690 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:18.899919 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:18.900069 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:18.937377 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:39575
I20250115 02:53:18.937484 10551 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:39575 every 8 connection(s)
I20250115 02:53:18.948372 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:18.958974 10552 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45789
I20250115 02:53:18.959383 10552 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:18.960212 10552 heartbeater.cc:509] Master 127.2.36.124:45789 requested a full tablet report, sending...
I20250115 02:53:18.963040 10402 ts_manager.cc:194] Registered new tserver with Master: fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575)
I20250115 02:53:18.966778 10553 heartbeater.cc:346] Connected to a master server at 127.2.36.125:40819
I20250115 02:53:18.967226 10553 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:18.967954 10554 heartbeater.cc:346] Connected to a master server at 127.2.36.126:40125
I20250115 02:53:18.968145 10553 heartbeater.cc:509] Master 127.2.36.125:40819 requested a full tablet report, sending...
I20250115 02:53:18.968428 10554 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:18.969367 10554 heartbeater.cc:509] Master 127.2.36.126:40125 requested a full tablet report, sending...
I20250115 02:53:18.971424 10337 ts_manager.cc:194] Registered new tserver with Master: fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575)
I20250115 02:53:18.972384 10272 ts_manager.cc:194] Registered new tserver with Master: fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575)
W20250115 02:53:18.975589 10560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:18.976264 10559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:18.977389 10562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:18.978238 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:18.978993 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:18.979177 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:18.979308 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909598979297 us; error 0 us; skew 500 ppm
I20250115 02:53:18.979772 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:18.982084 2193 webserver.cc:458] Webserver started at http://127.2.36.66:36953/ using document root <none> and password file <none>
I20250115 02:53:18.982506 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:18.982658 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:18.982903 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:18.984130 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "2ae6b15bbd244631849f856d84105bdb"
format_stamp: "Formatted at 2025-01-15 02:53:18 on dist-test-slave-1lv2"
I20250115 02:53:18.988865 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:53:18.992358 10567 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:18.993039 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250115 02:53:18.993320 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "2ae6b15bbd244631849f856d84105bdb"
format_stamp: "Formatted at 2025-01-15 02:53:18 on dist-test-slave-1lv2"
I20250115 02:53:18.993570 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:19.009022 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:19.010187 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:19.011807 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:19.014523 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:19.014814 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250115 02:53:19.015094 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:19.015235 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:19.059528 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:43407
I20250115 02:53:19.059629 10629 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:43407 every 8 connection(s)
I20250115 02:53:19.078214 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:19.088423 10630 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45789
I20250115 02:53:19.088893 10630 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:19.089725 10630 heartbeater.cc:509] Master 127.2.36.124:45789 requested a full tablet report, sending...
I20250115 02:53:19.092306 10402 ts_manager.cc:194] Registered new tserver with Master: 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:19.092612 10631 heartbeater.cc:346] Connected to a master server at 127.2.36.125:40819
I20250115 02:53:19.093084 10631 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:19.094058 10631 heartbeater.cc:509] Master 127.2.36.125:40819 requested a full tablet report, sending...
I20250115 02:53:19.096891 10337 ts_manager.cc:194] Registered new tserver with Master: 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:19.098083 10445 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:19.098567 10445 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
W20250115 02:53:19.099440 10637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:19.100665 10445 leader_election.cc:290] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31dc9f4ea6ab494e8a4c3dddb7776b9c (127.2.36.125:40819), 7aeaf33e08c04784847da73e162f05af (127.2.36.124:45789)
W20250115 02:53:19.101070 10638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:19.101833 10412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9be9a98525244abfa67175d705720235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aeaf33e08c04784847da73e162f05af" is_pre_election: true
I20250115 02:53:19.102609 10412 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9be9a98525244abfa67175d705720235 in term 0.
I20250115 02:53:19.104063 10256 leader_election.cc:304] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aeaf33e08c04784847da73e162f05af, 9be9a98525244abfa67175d705720235; no voters:
I20250115 02:53:19.105105 10347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9be9a98525244abfa67175d705720235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" is_pre_election: true
I20250115 02:53:19.105875 10347 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9be9a98525244abfa67175d705720235 in term 0.
I20250115 02:53:19.106168 10445 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:19.106693 10445 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:19.107110 10445 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:19.110677 10632 heartbeater.cc:346] Connected to a master server at 127.2.36.126:40125
I20250115 02:53:19.111096 10632 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:19.111903 10632 heartbeater.cc:509] Master 127.2.36.126:40125 requested a full tablet report, sending...
I20250115 02:53:19.116113 10445 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:19.118364 10445 leader_election.cc:290] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [CANDIDATE]: Term 1 election: Requested vote from peers 31dc9f4ea6ab494e8a4c3dddb7776b9c (127.2.36.125:40819), 7aeaf33e08c04784847da73e162f05af (127.2.36.124:45789)
I20250115 02:53:19.118983 10347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9be9a98525244abfa67175d705720235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c"
I20250115 02:53:19.119678 10347 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:19.119688 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:19.119940 10640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:19.120735 10412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9be9a98525244abfa67175d705720235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aeaf33e08c04784847da73e162f05af"
I20250115 02:53:19.121178 10412 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:19.122712 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:19.123045 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:19.123301 10271 ts_manager.cc:194] Registered new tserver with Master: 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:19.123453 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909599123431 us; error 0 us; skew 500 ppm
I20250115 02:53:19.125144 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:19.129176 10347 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9be9a98525244abfa67175d705720235 in term 1.
I20250115 02:53:19.130301 10258 leader_election.cc:304] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31dc9f4ea6ab494e8a4c3dddb7776b9c, 9be9a98525244abfa67175d705720235; no voters:
I20250115 02:53:19.131273 10445 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:19.131927 10412 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9be9a98525244abfa67175d705720235 in term 1.
I20250115 02:53:19.135185 10445 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 1 LEADER]: Becoming Leader. State: Replica: 9be9a98525244abfa67175d705720235, State: Running, Role: LEADER
I20250115 02:53:19.136171 10445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:19.141247 2193 webserver.cc:458] Webserver started at http://127.2.36.67:39021/ using document root <none> and password file <none>
I20250115 02:53:19.141775 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:19.141996 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:19.142304 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:19.143666 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "3f908969f8ee46978fd73c305c92b581"
format_stamp: "Formatted at 2025-01-15 02:53:19 on dist-test-slave-1lv2"
I20250115 02:53:19.143801 10643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9be9a98525244abfa67175d705720235. Latest consensus state: current_term: 1 leader_uuid: "9be9a98525244abfa67175d705720235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:19.144446 10643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:19.149531 10646 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:19.150220 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250115 02:53:19.155723 10648 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:19.156680 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250115 02:53:19.156973 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "3f908969f8ee46978fd73c305c92b581"
format_stamp: "Formatted at 2025-01-15 02:53:19 on dist-test-slave-1lv2"
I20250115 02:53:19.157217 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTableFilter.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:19.157812 10646 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:19.168897 10347 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 1 FOLLOWER]: Refusing update from remote peer 9be9a98525244abfa67175d705720235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:19.169066 10412 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 1 FOLLOWER]: Refusing update from remote peer 9be9a98525244abfa67175d705720235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:19.170670 10643 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:19.171484 10445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:19.185719 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:19.187283 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:19.192147 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:19.192445 10444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9be9a98525244abfa67175d705720235. Latest consensus state: current_term: 1 leader_uuid: "9be9a98525244abfa67175d705720235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:19.193346 10444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:19.196550 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:19.196954 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250115 02:53:19.197279 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:19.197569 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:19.197228 10469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9be9a98525244abfa67175d705720235. Latest consensus state: current_term: 1 leader_uuid: "9be9a98525244abfa67175d705720235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:19.199056 10469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:19.210211 10646 catalog_manager.cc:1348] Generated new cluster ID: 468d8bc4dac14121ad866951529e7fdc
I20250115 02:53:19.210579 10646 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:19.219751 10469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9be9a98525244abfa67175d705720235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:19.220690 10469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:19.221508 10643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9be9a98525244abfa67175d705720235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:19.223419 10643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:19.222540 10661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9be9a98525244abfa67175d705720235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:19.224040 10661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:19.231884 10444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9be9a98525244abfa67175d705720235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:19.232913 10444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:19.260465 10646 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:53:19.262169 10646 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:19.288609 10646 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235: Generated new TSK 0
I20250115 02:53:19.289306 10646 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:19.307796 10271 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44174
I20250115 02:53:19.308387 10272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44164
I20250115 02:53:19.321025 10632 heartbeater.cc:501] Master 127.2.36.126:40125 was elected leader, sending a full tablet report...
I20250115 02:53:19.322422 10554 heartbeater.cc:501] Master 127.2.36.126:40125 was elected leader, sending a full tablet report...
I20250115 02:53:19.332481 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:32945
I20250115 02:53:19.332651 10719 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:32945 every 8 connection(s)
I20250115 02:53:19.361465 10720 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45789
I20250115 02:53:19.362023 10720 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:19.363000 10720 heartbeater.cc:509] Master 127.2.36.124:45789 requested a full tablet report, sending...
I20250115 02:53:19.365772 10402 ts_manager.cc:194] Registered new tserver with Master: 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945)
I20250115 02:53:19.374934 10721 heartbeater.cc:346] Connected to a master server at 127.2.36.125:40819
I20250115 02:53:19.375882 10721 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:19.376690 10721 heartbeater.cc:509] Master 127.2.36.125:40819 requested a full tablet report, sending...
I20250115 02:53:19.380281 10337 ts_manager.cc:194] Registered new tserver with Master: 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945)
I20250115 02:53:19.384012 10723 heartbeater.cc:346] Connected to a master server at 127.2.36.126:40125
I20250115 02:53:19.384390 10723 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:19.385231 10723 heartbeater.cc:509] Master 127.2.36.126:40125 requested a full tablet report, sending...
I20250115 02:53:19.386962 10272 ts_manager.cc:194] Registered new tserver with Master: 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945)
I20250115 02:53:19.387238 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016139139s
I20250115 02:53:19.388643 10272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44182
I20250115 02:53:19.402511 10723 heartbeater.cc:501] Master 127.2.36.126:40125 was elected leader, sending a full tablet report...
I20250115 02:53:19.423422 10271 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44188:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:53:19.428947 10271 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:53:19.623883 10517 tablet_service.cc:1467] Processing CreateTablet for tablet 6de1916181484e67a6929c4b7668904d (DEFAULT_TABLE table=ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:19.627501 10517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6de1916181484e67a6929c4b7668904d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:19.634085 10685 tablet_service.cc:1467] Processing CreateTablet for tablet 6de1916181484e67a6929c4b7668904d (DEFAULT_TABLE table=ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:19.635854 10685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6de1916181484e67a6929c4b7668904d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:19.637996 10516 tablet_service.cc:1467] Processing CreateTablet for tablet c910d4473d284679b8a2841e01fb7b40 (DEFAULT_TABLE table=ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:19.638216 10515 tablet_service.cc:1467] Processing CreateTablet for tablet dd5854873b5c4fa3a570f29f704e6dcd (DEFAULT_TABLE table=ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:19.639642 10516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c910d4473d284679b8a2841e01fb7b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:19.647913 10515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5854873b5c4fa3a570f29f704e6dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:19.664330 10684 tablet_service.cc:1467] Processing CreateTablet for tablet c910d4473d284679b8a2841e01fb7b40 (DEFAULT_TABLE table=ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:19.664662 10683 tablet_service.cc:1467] Processing CreateTablet for tablet dd5854873b5c4fa3a570f29f704e6dcd (DEFAULT_TABLE table=ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:19.665768 10684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c910d4473d284679b8a2841e01fb7b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:19.666448 10683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5854873b5c4fa3a570f29f704e6dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:19.710462 10595 tablet_service.cc:1467] Processing CreateTablet for tablet 6de1916181484e67a6929c4b7668904d (DEFAULT_TABLE table=ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:19.710731 10594 tablet_service.cc:1467] Processing CreateTablet for tablet c910d4473d284679b8a2841e01fb7b40 (DEFAULT_TABLE table=ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:19.712029 10595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6de1916181484e67a6929c4b7668904d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:19.712977 10593 tablet_service.cc:1467] Processing CreateTablet for tablet dd5854873b5c4fa3a570f29f704e6dcd (DEFAULT_TABLE table=ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:19.714275 10593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5854873b5c4fa3a570f29f704e6dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:19.717730 10594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c910d4473d284679b8a2841e01fb7b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:19.763471 10743 tablet_bootstrap.cc:492] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb: Bootstrap starting.
I20250115 02:53:19.767366 10742 tablet_bootstrap.cc:492] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812: Bootstrap starting.
I20250115 02:53:19.781498 10742 tablet_bootstrap.cc:654] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:19.785928 10743 tablet_bootstrap.cc:654] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:19.795368 10741 tablet_bootstrap.cc:492] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581: Bootstrap starting.
I20250115 02:53:19.801440 10743 tablet_bootstrap.cc:492] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb: No bootstrap required, opened a new log
I20250115 02:53:19.802096 10743 ts_tablet_manager.cc:1397] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.004s
I20250115 02:53:19.807320 10743 raft_consensus.cc:357] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:19.808333 10743 raft_consensus.cc:383] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:19.808714 10743 raft_consensus.cc:738] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae6b15bbd244631849f856d84105bdb, State: Initialized, Role: FOLLOWER
I20250115 02:53:19.811460 10743 consensus_queue.cc:260] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:19.813905 10742 tablet_bootstrap.cc:492] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812: No bootstrap required, opened a new log
I20250115 02:53:19.814669 10742 ts_tablet_manager.cc:1397] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.004s
I20250115 02:53:19.814868 10741 tablet_bootstrap.cc:654] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:19.821892 10743 ts_tablet_manager.cc:1428] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb: Time spent starting tablet: real 0.019s user 0.006s sys 0.007s
I20250115 02:53:19.822149 10742 raft_consensus.cc:357] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:19.822993 10742 raft_consensus.cc:383] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:19.823280 10742 raft_consensus.cc:738] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb481af490f4be18851faed3b37b812, State: Initialized, Role: FOLLOWER
I20250115 02:53:19.826660 10742 consensus_queue.cc:260] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:19.832073 10743 tablet_bootstrap.cc:492] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb: Bootstrap starting.
I20250115 02:53:19.837868 10743 tablet_bootstrap.cc:654] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:19.847699 10742 ts_tablet_manager.cc:1428] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812: Time spent starting tablet: real 0.033s user 0.006s sys 0.003s
I20250115 02:53:19.851987 10468 catalog_manager.cc:1260] Loaded cluster ID: 468d8bc4dac14121ad866951529e7fdc
I20250115 02:53:19.852342 10468 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c: loading cluster ID for follower catalog manager: success
I20250115 02:53:19.857882 10742 tablet_bootstrap.cc:492] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812: Bootstrap starting.
I20250115 02:53:19.859894 10468 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c: acquiring CA information for follower catalog manager: success
I20250115 02:53:19.861516 10743 tablet_bootstrap.cc:492] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb: No bootstrap required, opened a new log
I20250115 02:53:19.862082 10743 ts_tablet_manager.cc:1397] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.004s
I20250115 02:53:19.864928 10746 raft_consensus.cc:491] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:19.865487 10746 raft_consensus.cc:513] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:19.865979 10468 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:19.866492 10480 catalog_manager.cc:1260] Loaded cluster ID: 468d8bc4dac14121ad866951529e7fdc
I20250115 02:53:19.867358 10480 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af: loading cluster ID for follower catalog manager: success
I20250115 02:53:19.869846 10746 leader_election.cc:290] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575), 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945)
I20250115 02:53:19.873183 10742 tablet_bootstrap.cc:654] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:19.852583 10741 tablet_bootstrap.cc:492] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581: No bootstrap required, opened a new log
I20250115 02:53:19.875922 10741 ts_tablet_manager.cc:1397] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581: Time spent bootstrapping tablet: real 0.081s user 0.014s sys 0.005s
I20250115 02:53:19.880165 10743 raft_consensus.cc:357] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:19.880404 10741 raft_consensus.cc:357] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:19.881145 10741 raft_consensus.cc:383] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:19.881453 10741 raft_consensus.cc:738] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f908969f8ee46978fd73c305c92b581, State: Initialized, Role: FOLLOWER
I20250115 02:53:19.887928 10747 raft_consensus.cc:491] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:19.881048 10743 raft_consensus.cc:383] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:19.888513 10747 raft_consensus.cc:513] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:19.891047 10743 raft_consensus.cc:738] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae6b15bbd244631849f856d84105bdb, State: Initialized, Role: FOLLOWER
I20250115 02:53:19.886555 10741 consensus_queue.cc:260] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:19.894387 10743 consensus_queue.cc:260] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:19.897450 10743 ts_tablet_manager.cc:1428] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb: Time spent starting tablet: real 0.035s user 0.008s sys 0.000s
I20250115 02:53:19.901813 10743 tablet_bootstrap.cc:492] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb: Bootstrap starting.
I20250115 02:53:19.898455 10480 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af: acquiring CA information for follower catalog manager: success
I20250115 02:53:19.898706 10747 leader_election.cc:290] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407), 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945)
I20250115 02:53:19.913614 10743 tablet_bootstrap.cc:654] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:19.931352 10742 tablet_bootstrap.cc:492] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812: No bootstrap required, opened a new log
I20250115 02:53:19.937899 10742 ts_tablet_manager.cc:1397] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812: Time spent bootstrapping tablet: real 0.080s user 0.013s sys 0.016s
I20250115 02:53:19.944793 10742 raft_consensus.cc:357] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:19.945676 10742 raft_consensus.cc:383] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:19.945987 10742 raft_consensus.cc:738] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb481af490f4be18851faed3b37b812, State: Initialized, Role: FOLLOWER
I20250115 02:53:19.946162 10480 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:19.946828 10742 consensus_queue.cc:260] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:19.949494 10742 ts_tablet_manager.cc:1428] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20250115 02:53:19.962504 10742 tablet_bootstrap.cc:492] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812: Bootstrap starting.
I20250115 02:53:19.970465 10695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "2ae6b15bbd244631849f856d84105bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f908969f8ee46978fd73c305c92b581" is_pre_election: true
I20250115 02:53:19.972164 10695 raft_consensus.cc:2463] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae6b15bbd244631849f856d84105bdb in term 0.
I20250115 02:53:19.973726 10568 leader_election.cc:304] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae6b15bbd244631849f856d84105bdb, 3f908969f8ee46978fd73c305c92b581; no voters:
I20250115 02:53:19.974643 10746 raft_consensus.cc:2798] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:19.975059 10746 raft_consensus.cc:491] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:19.975361 10746 raft_consensus.cc:3054] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:19.970928 10741 ts_tablet_manager.cc:1428] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581: Time spent starting tablet: real 0.095s user 0.014s sys 0.013s
I20250115 02:53:19.989532 10741 tablet_bootstrap.cc:492] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581: Bootstrap starting.
I20250115 02:53:19.997488 10742 tablet_bootstrap.cc:654] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:19.998037 10527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "2ae6b15bbd244631849f856d84105bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb481af490f4be18851faed3b37b812" is_pre_election: true
W20250115 02:53:20.003985 10569 leader_election.cc:343] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:20.023233 10605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c910d4473d284679b8a2841e01fb7b40" candidate_uuid: "fcb481af490f4be18851faed3b37b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb" is_pre_election: true
I20250115 02:53:20.024000 10605 raft_consensus.cc:2463] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcb481af490f4be18851faed3b37b812 in term 0.
I20250115 02:53:20.024786 10742 tablet_bootstrap.cc:492] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812: No bootstrap required, opened a new log
I20250115 02:53:20.025377 10742 ts_tablet_manager.cc:1397] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812: Time spent bootstrapping tablet: real 0.063s user 0.018s sys 0.004s
I20250115 02:53:20.026412 10743 tablet_bootstrap.cc:492] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb: No bootstrap required, opened a new log
I20250115 02:53:20.026961 10743 ts_tablet_manager.cc:1397] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb: Time spent bootstrapping tablet: real 0.125s user 0.024s sys 0.016s
I20250115 02:53:20.027506 10490 leader_election.cc:304] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae6b15bbd244631849f856d84105bdb, fcb481af490f4be18851faed3b37b812; no voters:
I20250115 02:53:20.028518 10747 raft_consensus.cc:2798] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:20.028507 10742 raft_consensus.cc:357] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:20.029134 10747 raft_consensus.cc:491] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:20.029608 10742 raft_consensus.cc:383] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:20.029699 10747 raft_consensus.cc:3054] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:20.030145 10742 raft_consensus.cc:738] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb481af490f4be18851faed3b37b812, State: Initialized, Role: FOLLOWER
I20250115 02:53:20.031661 10741 tablet_bootstrap.cc:654] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:20.029708 10743 raft_consensus.cc:357] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:20.035943 10743 raft_consensus.cc:383] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:20.036240 10743 raft_consensus.cc:738] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae6b15bbd244631849f856d84105bdb, State: Initialized, Role: FOLLOWER
I20250115 02:53:20.036983 10743 consensus_queue.cc:260] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:20.038707 10747 raft_consensus.cc:513] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:20.041725 10747 leader_election.cc:290] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407), 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945)
I20250115 02:53:20.039635 10742 consensus_queue.cc:260] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:20.042723 10605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c910d4473d284679b8a2841e01fb7b40" candidate_uuid: "fcb481af490f4be18851faed3b37b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb"
I20250115 02:53:20.043370 10605 raft_consensus.cc:3054] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:20.044641 10742 ts_tablet_manager.cc:1428] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20250115 02:53:20.051414 10694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c910d4473d284679b8a2841e01fb7b40" candidate_uuid: "fcb481af490f4be18851faed3b37b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f908969f8ee46978fd73c305c92b581"
I20250115 02:53:20.053225 10695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c910d4473d284679b8a2841e01fb7b40" candidate_uuid: "fcb481af490f4be18851faed3b37b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f908969f8ee46978fd73c305c92b581" is_pre_election: true
I20250115 02:53:20.057087 10759 raft_consensus.cc:491] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:20.057647 10759 raft_consensus.cc:513] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
W20250115 02:53:20.059744 10490 leader_election.cc:343] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:20.060482 10743 ts_tablet_manager.cc:1428] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb: Time spent starting tablet: real 0.033s user 0.010s sys 0.000s
I20250115 02:53:20.060586 10527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5854873b5c4fa3a570f29f704e6dcd" candidate_uuid: "2ae6b15bbd244631849f856d84105bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb481af490f4be18851faed3b37b812" is_pre_election: true
I20250115 02:53:20.061286 10527 raft_consensus.cc:2463] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae6b15bbd244631849f856d84105bdb in term 0.
I20250115 02:53:20.062332 10759 leader_election.cc:290] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575), 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945)
I20250115 02:53:20.062367 10569 leader_election.cc:304] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae6b15bbd244631849f856d84105bdb, fcb481af490f4be18851faed3b37b812; no voters:
I20250115 02:53:20.063375 10759 raft_consensus.cc:2798] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:20.063686 10759 raft_consensus.cc:491] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:20.063987 10759 raft_consensus.cc:3054] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:20.065434 10695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5854873b5c4fa3a570f29f704e6dcd" candidate_uuid: "2ae6b15bbd244631849f856d84105bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f908969f8ee46978fd73c305c92b581" is_pre_election: true
W20250115 02:53:20.067569 10568 leader_election.cc:343] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:20.067730 10746 raft_consensus.cc:513] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
W20250115 02:53:20.060493 10490 leader_election.cc:343] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:20.070402 10527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "2ae6b15bbd244631849f856d84105bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb481af490f4be18851faed3b37b812"
I20250115 02:53:20.071103 10759 raft_consensus.cc:513] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:20.071039 10527 raft_consensus.cc:3054] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:20.073155 10605 raft_consensus.cc:2463] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcb481af490f4be18851faed3b37b812 in term 1.
I20250115 02:53:20.074206 10695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5854873b5c4fa3a570f29f704e6dcd" candidate_uuid: "2ae6b15bbd244631849f856d84105bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f908969f8ee46978fd73c305c92b581"
I20250115 02:53:20.074440 10490 leader_election.cc:304] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ae6b15bbd244631849f856d84105bdb, fcb481af490f4be18851faed3b37b812; no voters: 3f908969f8ee46978fd73c305c92b581
I20250115 02:53:20.075333 10747 raft_consensus.cc:2798] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Leader election won for term 1
W20250115 02:53:20.075750 10568 leader_election.cc:343] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:20.075834 10747 raft_consensus.cc:695] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 1 LEADER]: Becoming Leader. State: Replica: fcb481af490f4be18851faed3b37b812, State: Running, Role: LEADER
I20250115 02:53:20.076079 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5854873b5c4fa3a570f29f704e6dcd" candidate_uuid: "2ae6b15bbd244631849f856d84105bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb481af490f4be18851faed3b37b812"
I20250115 02:53:20.076285 10759 leader_election.cc:290] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 1 election: Requested vote from peers fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575), 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945)
I20250115 02:53:20.076867 10747 consensus_queue.cc:237] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:20.076748 10526 raft_consensus.cc:3054] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:20.084859 10527 raft_consensus.cc:2463] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae6b15bbd244631849f856d84105bdb in term 1.
I20250115 02:53:20.086082 10569 leader_election.cc:304] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae6b15bbd244631849f856d84105bdb, fcb481af490f4be18851faed3b37b812; no voters:
I20250115 02:53:20.087031 10759 raft_consensus.cc:2798] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:20.087559 10759 raft_consensus.cc:695] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 1 LEADER]: Becoming Leader. State: Replica: 2ae6b15bbd244631849f856d84105bdb, State: Running, Role: LEADER
I20250115 02:53:20.088470 10759 consensus_queue.cc:237] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:20.092619 10526 raft_consensus.cc:2463] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae6b15bbd244631849f856d84105bdb in term 1.
I20250115 02:53:20.093984 10569 leader_election.cc:304] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ae6b15bbd244631849f856d84105bdb, fcb481af490f4be18851faed3b37b812; no voters: 3f908969f8ee46978fd73c305c92b581
I20250115 02:53:20.097617 10759 raft_consensus.cc:2798] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:20.098100 10759 raft_consensus.cc:695] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 1 LEADER]: Becoming Leader. State: Replica: 2ae6b15bbd244631849f856d84105bdb, State: Running, Role: LEADER
I20250115 02:53:20.100661 10759 consensus_queue.cc:237] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:20.102922 10271 catalog_manager.cc:5526] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ae6b15bbd244631849f856d84105bdb (127.2.36.66). New cstate: current_term: 1 leader_uuid: "2ae6b15bbd244631849f856d84105bdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } health_report { overall_health: HEALTHY } } }
I20250115 02:53:20.108481 10695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "2ae6b15bbd244631849f856d84105bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f908969f8ee46978fd73c305c92b581"
I20250115 02:53:20.109225 10695 raft_consensus.cc:3054] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:20.113590 10746 leader_election.cc:290] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 1 election: Requested vote from peers fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575), 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945)
I20250115 02:53:20.122840 10695 raft_consensus.cc:2463] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae6b15bbd244631849f856d84105bdb in term 1.
I20250115 02:53:20.130906 10741 tablet_bootstrap.cc:492] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581: No bootstrap required, opened a new log
I20250115 02:53:20.131523 10741 ts_tablet_manager.cc:1397] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581: Time spent bootstrapping tablet: real 0.147s user 0.024s sys 0.002s
I20250115 02:53:20.137526 10741 raft_consensus.cc:357] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:20.138355 10741 raft_consensus.cc:383] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:20.138702 10741 raft_consensus.cc:738] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f908969f8ee46978fd73c305c92b581, State: Initialized, Role: FOLLOWER
I20250115 02:53:20.139562 10741 consensus_queue.cc:260] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:20.142278 10741 ts_tablet_manager.cc:1428] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250115 02:53:20.148635 10741 tablet_bootstrap.cc:492] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581: Bootstrap starting.
I20250115 02:53:20.155088 10741 tablet_bootstrap.cc:654] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:20.174705 10272 catalog_manager.cc:5526] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcb481af490f4be18851faed3b37b812 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "fcb481af490f4be18851faed3b37b812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:20.201267 10270 catalog_manager.cc:5526] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ae6b15bbd244631849f856d84105bdb (127.2.36.66). New cstate: current_term: 1 leader_uuid: "2ae6b15bbd244631849f856d84105bdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:20.215945 10741 tablet_bootstrap.cc:492] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581: No bootstrap required, opened a new log
I20250115 02:53:20.216490 10741 ts_tablet_manager.cc:1397] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581: Time spent bootstrapping tablet: real 0.068s user 0.016s sys 0.005s
I20250115 02:53:20.219286 10741 raft_consensus.cc:357] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:20.220083 10741 raft_consensus.cc:383] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:20.220360 10741 raft_consensus.cc:738] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f908969f8ee46978fd73c305c92b581, State: Initialized, Role: FOLLOWER
I20250115 02:53:20.221232 10741 consensus_queue.cc:260] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:20.236693 10741 ts_tablet_manager.cc:1428] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20250115 02:53:20.280267 2193 ksck_remote-test.cc:683] Generating row writes...
I20250115 02:53:20.351365 10754 raft_consensus.cc:491] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:20.351904 10754 raft_consensus.cc:513] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:20.357165 10754 leader_election.cc:290] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575), 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:20.401294 10605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c910d4473d284679b8a2841e01fb7b40" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb" is_pre_election: true
I20250115 02:53:20.402043 10605 raft_consensus.cc:2388] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f908969f8ee46978fd73c305c92b581 in current term 1: Already voted for candidate fcb481af490f4be18851faed3b37b812 in this term.
I20250115 02:53:20.410085 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c910d4473d284679b8a2841e01fb7b40" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb481af490f4be18851faed3b37b812" is_pre_election: true
I20250115 02:53:20.411758 10650 leader_election.cc:304] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f908969f8ee46978fd73c305c92b581; no voters: 2ae6b15bbd244631849f856d84105bdb, fcb481af490f4be18851faed3b37b812
I20250115 02:53:20.412979 10754 raft_consensus.cc:3054] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:20.425839 10754 raft_consensus.cc:2743] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:53:20.501436 10747 consensus_queue.cc:1035] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:20.526114 10747 consensus_queue.cc:1035] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:20.535254 10526 raft_consensus.cc:1270] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Refusing update from remote peer 2ae6b15bbd244631849f856d84105bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:20.536583 10694 raft_consensus.cc:1270] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Refusing update from remote peer 2ae6b15bbd244631849f856d84105bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250115 02:53:20.593564 10569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.36.65:39575, user_credentials={real_user=slave}} blocked reactor thread for 55800.5us
W20250115 02:53:20.593462 10568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.36.67:32945, user_credentials={real_user=slave}} blocked reactor thread for 51484.1us
I20250115 02:53:20.593479 10746 consensus_queue.cc:1035] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 02:53:20.595386 10759 consensus_queue.cc:1035] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:20.625672 10527 raft_consensus.cc:1270] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Refusing update from remote peer 2ae6b15bbd244631849f856d84105bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:20.632030 10746 consensus_queue.cc:1035] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:20.641925 10694 raft_consensus.cc:3054] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:20.658178 10694 raft_consensus.cc:1270] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Refusing update from remote peer 2ae6b15bbd244631849f856d84105bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:20.680277 10759 consensus_queue.cc:1035] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:20.727135 10803 mvcc.cc:204] Tried to move back new op lower bound from 7114381723763970048 to 7114381721972477952. Current Snapshot: MvccSnapshot[applied={T|T < 7114381723763970048}]
I20250115 02:53:22.137020 10812 raft_consensus.cc:491] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2ae6b15bbd244631849f856d84105bdb)
I20250115 02:53:22.137678 10812 raft_consensus.cc:513] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:22.143419 10812 leader_election.cc:290] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945), 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:22.290473 10812 raft_consensus.cc:491] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2ae6b15bbd244631849f856d84105bdb)
I20250115 02:53:22.291086 10812 raft_consensus.cc:513] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:22.308280 10812 leader_election.cc:290] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407), 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945)
I20250115 02:53:22.429415 10814 raft_consensus.cc:491] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2ae6b15bbd244631849f856d84105bdb)
I20250115 02:53:22.430030 10814 raft_consensus.cc:513] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:22.432127 10814 leader_election.cc:290] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575), 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:22.437083 10815 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9be9a98525244abfa67175d705720235)
I20250115 02:53:22.437621 10815 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:22.439685 10815 leader_election.cc:290] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9be9a98525244abfa67175d705720235 (127.2.36.126:40125), 31dc9f4ea6ab494e8a4c3dddb7776b9c (127.2.36.125:40819)
I20250115 02:53:22.441360 10282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7aeaf33e08c04784847da73e162f05af" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9be9a98525244abfa67175d705720235" is_pre_election: true
I20250115 02:53:22.445195 10603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5854873b5c4fa3a570f29f704e6dcd" candidate_uuid: "fcb481af490f4be18851faed3b37b812" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb" is_pre_election: true
I20250115 02:53:22.450703 10814 raft_consensus.cc:491] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fcb481af490f4be18851faed3b37b812)
I20250115 02:53:22.451376 10814 raft_consensus.cc:513] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:22.453723 10814 leader_election.cc:290] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575), 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:22.458638 10603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c910d4473d284679b8a2841e01fb7b40" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb" is_pre_election: true
I20250115 02:53:22.575342 10813 raft_consensus.cc:491] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fcb481af490f4be18851faed3b37b812)
I20250115 02:53:22.575968 10813 raft_consensus.cc:513] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:22.578130 10813 leader_election.cc:290] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575), 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945)
I20250115 02:53:22.579226 10814 raft_consensus.cc:491] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2ae6b15bbd244631849f856d84105bdb)
I20250115 02:53:22.579785 10814 raft_consensus.cc:513] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } }
I20250115 02:53:22.580253 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c910d4473d284679b8a2841e01fb7b40" candidate_uuid: "2ae6b15bbd244631849f856d84105bdb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fcb481af490f4be18851faed3b37b812" is_pre_election: true
I20250115 02:53:22.581614 10814 leader_election.cc:290] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575), 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:22.583379 10603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5854873b5c4fa3a570f29f704e6dcd" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb" is_pre_election: true
I20250115 02:53:22.648887 10272 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44182 (request call id 77) took 1695 ms. Trace:
I20250115 02:53:22.649346 10272 rpcz_store.cc:276] 0115 02:53:20.953887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:22.647619 (+1693732us) service_pool.cc:225] Handling call
0115 02:53:22.648824 (+ 1205us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":0}
I20250115 02:53:22.654733 10602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb" is_pre_election: true
I20250115 02:53:22.686137 10335 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36372 (request call id 106) took 1732 ms. Trace:
I20250115 02:53:22.694986 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fcb481af490f4be18851faed3b37b812" is_pre_election: true
I20250115 02:53:22.695740 10526 raft_consensus.cc:2463] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f908969f8ee46978fd73c305c92b581 in term 1.
I20250115 02:53:22.697479 10650 leader_election.cc:304] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f908969f8ee46978fd73c305c92b581, fcb481af490f4be18851faed3b37b812; no voters: 2ae6b15bbd244631849f856d84105bdb
I20250115 02:53:22.698280 10814 raft_consensus.cc:2798] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:53:22.698652 10525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5854873b5c4fa3a570f29f704e6dcd" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fcb481af490f4be18851faed3b37b812" is_pre_election: true
I20250115 02:53:22.699771 10692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "fcb481af490f4be18851faed3b37b812" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f908969f8ee46978fd73c305c92b581" is_pre_election: true
I20250115 02:53:22.700382 10267 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44164 (request call id 88) took 1727 ms. Trace:
I20250115 02:53:22.698650 10814 raft_consensus.cc:491] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2ae6b15bbd244631849f856d84105bdb)
I20250115 02:53:22.700776 10267 rpcz_store.cc:276] 0115 02:53:20.972610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:21.056694 (+ 84084us) service_pool.cc:225] Handling call
0115 02:53:22.697109 (+1640415us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b5400199f10. stack: 00007f5a18a69f63 00007f5a18a69e60 00007f5a183bb584 00007f5a2242cff4 00007f5a2242cf89 00007f5a188df911 00007f5a18a5e991 00007f5a18a5eed2 00007f5a21535779 00007f5a214cde06 00007f5a2149eca8 00007f5a21596dee 00007f5a1fbfefc4 00007f5a1fbfef52 00007f5a1fbfee81 00007f5a1fbfedfc
0115 02:53:22.700190 (+ 3081us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":0,"spinlock_wait_cycles":522532608}
I20250115 02:53:22.701478 10692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5854873b5c4fa3a570f29f704e6dcd" candidate_uuid: "fcb481af490f4be18851faed3b37b812" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f908969f8ee46978fd73c305c92b581" is_pre_election: true
I20250115 02:53:22.701745 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c910d4473d284679b8a2841e01fb7b40" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fcb481af490f4be18851faed3b37b812" is_pre_election: true
I20250115 02:53:22.701009 10814 raft_consensus.cc:3054] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:22.703862 10693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c910d4473d284679b8a2841e01fb7b40" candidate_uuid: "2ae6b15bbd244631849f856d84105bdb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f908969f8ee46978fd73c305c92b581" is_pre_election: true
I20250115 02:53:22.706684 10490 leader_election.cc:304] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fcb481af490f4be18851faed3b37b812; no voters: 2ae6b15bbd244631849f856d84105bdb, 3f908969f8ee46978fd73c305c92b581
I20250115 02:53:22.707496 10812 raft_consensus.cc:2743] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 02:53:22.710577 10814 raft_consensus.cc:513] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:22.686612 10335 rpcz_store.cc:276] 0115 02:53:20.954083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:20.974889 (+ 20806us) service_pool.cc:225] Handling call
0115 02:53:22.684811 (+1709922us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540042a810. stack: 00007f5a18a69f63 00007f5a18a69e60 00007f5a183bb584 00007f5a2242cff4 00007f5a2242cf89 00007f5a188df911 00007f5a18a5e991 00007f5a18a5eed2 00007f5a21535779 00007f5a214cde06 00007f5a2149eca8 00007f5a21596dee 00007f5a1fbfefc4 00007f5a1fbfef52 00007f5a1fbfee81 00007f5a1fbfedfc
0115 02:53:22.685913 (+ 1102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3016587136}
I20250115 02:53:22.718626 10814 leader_election.cc:290] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 2 election: Requested vote from peers fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575), 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:22.719028 10568 leader_election.cc:304] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ae6b15bbd244631849f856d84105bdb; no voters: 3f908969f8ee46978fd73c305c92b581, fcb481af490f4be18851faed3b37b812
I20250115 02:53:22.721057 10813 raft_consensus.cc:2743] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 02:53:22.722193 10605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb"
I20250115 02:53:22.722893 10604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "fcb481af490f4be18851faed3b37b812" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb" is_pre_election: true
I20250115 02:53:22.724315 10490 leader_election.cc:304] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fcb481af490f4be18851faed3b37b812; no voters: 2ae6b15bbd244631849f856d84105bdb, 3f908969f8ee46978fd73c305c92b581
I20250115 02:53:22.724941 10816 raft_consensus.cc:491] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9be9a98525244abfa67175d705720235)
I20250115 02:53:22.725147 10812 raft_consensus.cc:2743] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 02:53:22.725468 10816 raft_consensus.cc:513] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:22.727648 10816 leader_election.cc:290] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9be9a98525244abfa67175d705720235 (127.2.36.126:40125), 7aeaf33e08c04784847da73e162f05af (127.2.36.124:45789)
I20250115 02:53:22.728750 10282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9be9a98525244abfa67175d705720235" is_pre_election: true
I20250115 02:53:22.732920 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fcb481af490f4be18851faed3b37b812"
I20250115 02:53:22.733048 10347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7aeaf33e08c04784847da73e162f05af" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" is_pre_election: true
I20250115 02:53:22.733547 10412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "7aeaf33e08c04784847da73e162f05af" is_pre_election: true
I20250115 02:53:22.733747 10347 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aeaf33e08c04784847da73e162f05af in term 1.
I20250115 02:53:22.734185 10412 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31dc9f4ea6ab494e8a4c3dddb7776b9c in term 1.
I20250115 02:53:22.734536 10650 leader_election.cc:304] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f908969f8ee46978fd73c305c92b581; no voters: 2ae6b15bbd244631849f856d84105bdb, fcb481af490f4be18851faed3b37b812
I20250115 02:53:22.736300 10814 raft_consensus.cc:2743] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 02:53:22.737391 10650 leader_election.cc:304] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f908969f8ee46978fd73c305c92b581; no voters: 2ae6b15bbd244631849f856d84105bdb, fcb481af490f4be18851faed3b37b812
I20250115 02:53:22.738158 10814 raft_consensus.cc:2743] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 02:53:22.739282 10650 leader_election.cc:304] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f908969f8ee46978fd73c305c92b581; no voters: 2ae6b15bbd244631849f856d84105bdb, fcb481af490f4be18851faed3b37b812
I20250115 02:53:22.740391 10814 raft_consensus.cc:2743] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250115 02:53:22.740370 10387 leader_election.cc:304] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31dc9f4ea6ab494e8a4c3dddb7776b9c, 7aeaf33e08c04784847da73e162f05af; no voters: 9be9a98525244abfa67175d705720235
I20250115 02:53:22.741518 10815 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:53:22.741887 10815 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9be9a98525244abfa67175d705720235)
I20250115 02:53:22.742153 10815 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:22.748205 10321 leader_election.cc:304] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31dc9f4ea6ab494e8a4c3dddb7776b9c, 7aeaf33e08c04784847da73e162f05af; no voters: 9be9a98525244abfa67175d705720235
I20250115 02:53:22.749094 10815 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:22.753012 10816 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:53:22.753484 10816 raft_consensus.cc:491] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9be9a98525244abfa67175d705720235)
I20250115 02:53:22.753937 10816 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:22.761152 10815 leader_election.cc:290] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [CANDIDATE]: Term 2 election: Requested vote from peers 9be9a98525244abfa67175d705720235 (127.2.36.126:40125), 31dc9f4ea6ab494e8a4c3dddb7776b9c (127.2.36.125:40819)
I20250115 02:53:22.762281 10346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7aeaf33e08c04784847da73e162f05af" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c"
I20250115 02:53:22.765972 10432 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:37268 (request call id 2) took 1944 ms. Trace:
I20250115 02:53:22.766379 10432 rpcz_store.cc:276] 0115 02:53:20.821506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:20.821664 (+ 158us) service_pool.cc:225] Handling call
0115 02:53:22.765765 (+1944101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 02:53:22.767576 10816 raft_consensus.cc:513] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:22.775345 10282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7aeaf33e08c04784847da73e162f05af" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9be9a98525244abfa67175d705720235"
I20250115 02:53:22.776219 10816 leader_election.cc:290] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [CANDIDATE]: Term 2 election: Requested vote from peers 9be9a98525244abfa67175d705720235 (127.2.36.126:40125), 7aeaf33e08c04784847da73e162f05af (127.2.36.124:45789)
I20250115 02:53:22.776718 10281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9be9a98525244abfa67175d705720235"
I20250115 02:53:22.783135 10389 leader_election.cc:304] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7aeaf33e08c04784847da73e162f05af; no voters: 31dc9f4ea6ab494e8a4c3dddb7776b9c, 9be9a98525244abfa67175d705720235
I20250115 02:53:22.783416 10411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "7aeaf33e08c04784847da73e162f05af"
I20250115 02:53:22.784314 10815 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250115 02:53:22.787428 10321 leader_election.cc:304] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31dc9f4ea6ab494e8a4c3dddb7776b9c; no voters: 7aeaf33e08c04784847da73e162f05af, 9be9a98525244abfa67175d705720235
I20250115 02:53:22.796967 10366 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:36402 (request call id 3) took 1843 ms. Trace:
I20250115 02:53:22.797389 10366 rpcz_store.cc:276] 0115 02:53:20.953630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:20.954122 (+ 492us) service_pool.cc:225] Handling call
0115 02:53:22.796713 (+1842591us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 02:53:22.805732 10816 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250115 02:53:22.820482 10368 debug-util.cc:398] Leaking SignalData structure 0x7b0800223d60 after lost signal to thread 2196
W20250115 02:53:22.821635 10368 debug-util.cc:398] Leaking SignalData structure 0x7b0800128ca0 after lost signal to thread 10306
W20250115 02:53:22.822656 10368 debug-util.cc:398] Leaking SignalData structure 0x7b080011a200 after lost signal to thread 10371
W20250115 02:53:22.828819 10368 debug-util.cc:398] Leaking SignalData structure 0x7b080004b6c0 after lost signal to thread 10436
W20250115 02:53:22.831241 10368 debug-util.cc:398] Leaking SignalData structure 0x7b08001bd340 after lost signal to thread 10551
W20250115 02:53:22.832479 10368 debug-util.cc:398] Leaking SignalData structure 0x7b0800158940 after lost signal to thread 10629
W20250115 02:53:22.833688 10368 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6de0 after lost signal to thread 10719
W20250115 02:53:22.834318 10368 debug-util.cc:398] Leaking SignalData structure 0x7b08000cd1a0 after lost signal to thread 10802
I20250115 02:53:22.833709 10302 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:44194 (request call id 2) took 2006 ms. Trace:
I20250115 02:53:22.836148 10302 rpcz_store.cc:276] 0115 02:53:20.826932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:20.827120 (+ 188us) service_pool.cc:225] Handling call
0115 02:53:22.833494 (+2006374us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 02:53:22.878489 10693 raft_consensus.cc:1235] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 2 FOLLOWER]: Rejecting Update request from peer 2ae6b15bbd244631849f856d84105bdb for earlier term 1. Current term is 2. Ops: []
I20250115 02:53:22.880801 10813 consensus_queue.cc:1046] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250115 02:53:22.928674 10411 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 2 FOLLOWER]: Rejecting Update request from peer 9be9a98525244abfa67175d705720235 for earlier term 1. Current term is 2. Ops: []
I20250115 02:53:22.930627 10768 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 }, Status: INVALID_TERM, Last received: 1.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250115 02:53:22.931630 10768 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 1 LEADER]: Stepping down as leader of term 1
I20250115 02:53:22.931929 10768 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9be9a98525244abfa67175d705720235, State: Running, Role: LEADER
I20250115 02:53:22.932653 10768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:22.938349 10768 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:22.945537 10819 raft_consensus.cc:3049] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 1 LEADER]: Stepping down as leader of term 1
I20250115 02:53:22.954707 10819 raft_consensus.cc:738] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2ae6b15bbd244631849f856d84105bdb, State: Running, Role: LEADER
I20250115 02:53:22.955927 10819 consensus_queue.cc:260] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:22.957268 10819 raft_consensus.cc:3054] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:23.154668 10464 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:24.276866 10812 raft_consensus.cc:491] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2ae6b15bbd244631849f856d84105bdb)
I20250115 02:53:24.277423 10812 raft_consensus.cc:513] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:24.279346 10812 leader_election.cc:290] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945), 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:24.280035 10693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "fcb481af490f4be18851faed3b37b812" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f908969f8ee46978fd73c305c92b581" is_pre_election: true
I20250115 02:53:24.280814 10693 raft_consensus.cc:2388] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fcb481af490f4be18851faed3b37b812 in current term 2: Already voted for candidate 3f908969f8ee46978fd73c305c92b581 in this term.
I20250115 02:53:24.282581 10605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "fcb481af490f4be18851faed3b37b812" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb" is_pre_election: true
I20250115 02:53:24.283321 10605 raft_consensus.cc:2463] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcb481af490f4be18851faed3b37b812 in term 2.
I20250115 02:53:24.284782 10490 leader_election.cc:304] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae6b15bbd244631849f856d84105bdb, fcb481af490f4be18851faed3b37b812; no voters:
I20250115 02:53:24.291311 10812 raft_consensus.cc:2798] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:53:24.291703 10812 raft_consensus.cc:491] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2ae6b15bbd244631849f856d84105bdb)
I20250115 02:53:24.292155 10812 raft_consensus.cc:3054] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:24.301695 10812 raft_consensus.cc:513] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:24.306878 10693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "fcb481af490f4be18851faed3b37b812" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f908969f8ee46978fd73c305c92b581"
I20250115 02:53:24.307606 10693 raft_consensus.cc:2388] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fcb481af490f4be18851faed3b37b812 in current term 2: Already voted for candidate 3f908969f8ee46978fd73c305c92b581 in this term.
I20250115 02:53:24.313194 10812 leader_election.cc:290] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 2 election: Requested vote from peers 3f908969f8ee46978fd73c305c92b581 (127.2.36.67:32945), 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:24.314618 10605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "fcb481af490f4be18851faed3b37b812" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb"
I20250115 02:53:24.322358 10605 raft_consensus.cc:2463] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcb481af490f4be18851faed3b37b812 in term 2.
I20250115 02:53:24.326694 10490 leader_election.cc:304] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ae6b15bbd244631849f856d84105bdb, fcb481af490f4be18851faed3b37b812; no voters: 3f908969f8ee46978fd73c305c92b581
I20250115 02:53:24.328940 10812 raft_consensus.cc:2798] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 2 FOLLOWER]: Leader election won for term 2
I20250115 02:53:24.332475 10812 raft_consensus.cc:695] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 2 LEADER]: Becoming Leader. State: Replica: fcb481af490f4be18851faed3b37b812, State: Running, Role: LEADER
I20250115 02:53:24.322789 10831 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:24.333973 10812 consensus_queue.cc:237] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:24.334204 10831 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:24.336356 10831 leader_election.cc:290] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9be9a98525244abfa67175d705720235 (127.2.36.126:40125), 31dc9f4ea6ab494e8a4c3dddb7776b9c (127.2.36.125:40819)
I20250115 02:53:24.339429 10346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7aeaf33e08c04784847da73e162f05af" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" is_pre_election: true
I20250115 02:53:24.339843 10281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7aeaf33e08c04784847da73e162f05af" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9be9a98525244abfa67175d705720235" is_pre_election: true
I20250115 02:53:24.340145 10346 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aeaf33e08c04784847da73e162f05af in term 2.
I20250115 02:53:24.340613 10281 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aeaf33e08c04784847da73e162f05af in term 2.
I20250115 02:53:24.342092 10387 leader_election.cc:304] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31dc9f4ea6ab494e8a4c3dddb7776b9c, 7aeaf33e08c04784847da73e162f05af; no voters:
I20250115 02:53:24.358906 10833 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:24.359409 10831 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250115 02:53:24.363795 10831 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:24.363543 10833 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:24.364214 10831 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:53:24.366539 10833 leader_election.cc:290] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31dc9f4ea6ab494e8a4c3dddb7776b9c (127.2.36.125:40819), 7aeaf33e08c04784847da73e162f05af (127.2.36.124:45789)
I20250115 02:53:24.367828 10411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9be9a98525244abfa67175d705720235" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "7aeaf33e08c04784847da73e162f05af" is_pre_election: true
I20250115 02:53:24.367828 10346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9be9a98525244abfa67175d705720235" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" is_pre_election: true
I20250115 02:53:24.369237 10346 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9be9a98525244abfa67175d705720235 in term 2.
I20250115 02:53:24.374905 10258 leader_election.cc:304] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31dc9f4ea6ab494e8a4c3dddb7776b9c, 9be9a98525244abfa67175d705720235; no voters:
I20250115 02:53:24.375531 10833 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250115 02:53:24.375946 10833 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:24.376329 10833 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:53:24.380290 10831 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:24.384727 10411 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9be9a98525244abfa67175d705720235 in current term 3: Already voted for candidate 7aeaf33e08c04784847da73e162f05af in this term.
I20250115 02:53:24.385072 10346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7aeaf33e08c04784847da73e162f05af" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c"
I20250115 02:53:24.385715 10346 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:53:24.390419 10281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7aeaf33e08c04784847da73e162f05af" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9be9a98525244abfa67175d705720235"
I20250115 02:53:24.395905 10346 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7aeaf33e08c04784847da73e162f05af in term 3.
I20250115 02:53:24.396168 10833 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:24.398147 10833 leader_election.cc:290] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [CANDIDATE]: Term 3 election: Requested vote from peers 31dc9f4ea6ab494e8a4c3dddb7776b9c (127.2.36.125:40819), 7aeaf33e08c04784847da73e162f05af (127.2.36.124:45789)
I20250115 02:53:24.403774 10411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9be9a98525244abfa67175d705720235" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "7aeaf33e08c04784847da73e162f05af"
I20250115 02:53:24.404716 10411 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9be9a98525244abfa67175d705720235 in current term 3: Already voted for candidate 7aeaf33e08c04784847da73e162f05af in this term.
I20250115 02:53:24.406070 10831 leader_election.cc:290] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [CANDIDATE]: Term 3 election: Requested vote from peers 9be9a98525244abfa67175d705720235 (127.2.36.126:40125), 31dc9f4ea6ab494e8a4c3dddb7776b9c (127.2.36.125:40819)
I20250115 02:53:24.403846 10281 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7aeaf33e08c04784847da73e162f05af in current term 3: Already voted for candidate 9be9a98525244abfa67175d705720235 in this term.
I20250115 02:53:24.407068 10387 leader_election.cc:304] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31dc9f4ea6ab494e8a4c3dddb7776b9c, 7aeaf33e08c04784847da73e162f05af; no voters:
I20250115 02:53:24.408515 10346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9be9a98525244abfa67175d705720235" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c"
I20250115 02:53:24.409330 10346 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9be9a98525244abfa67175d705720235 in current term 3: Already voted for candidate 7aeaf33e08c04784847da73e162f05af in this term.
I20250115 02:53:24.410858 10831 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 3 FOLLOWER]: Leader election won for term 3
I20250115 02:53:24.410858 10258 leader_election.cc:304] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9be9a98525244abfa67175d705720235; no voters: 31dc9f4ea6ab494e8a4c3dddb7776b9c, 7aeaf33e08c04784847da73e162f05af
I20250115 02:53:24.411711 10833 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250115 02:53:24.412945 10831 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 3 LEADER]: Becoming Leader. State: Replica: 7aeaf33e08c04784847da73e162f05af, State: Running, Role: LEADER
I20250115 02:53:24.413748 10831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } }
I20250115 02:53:24.429947 10835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aeaf33e08c04784847da73e162f05af. Latest consensus state: current_term: 3 leader_uuid: "7aeaf33e08c04784847da73e162f05af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:24.430774 10835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:24.432611 10837 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:24.446864 10837 catalog_manager.cc:670] Loaded metadata for table ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d]
I20250115 02:53:24.461577 10837 tablet_loader.cc:96] loaded metadata for tablet 6de1916181484e67a6929c4b7668904d (table ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d])
I20250115 02:53:24.465092 10837 tablet_loader.cc:96] loaded metadata for tablet c910d4473d284679b8a2841e01fb7b40 (table ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d])
I20250115 02:53:24.466485 10837 tablet_loader.cc:96] loaded metadata for tablet dd5854873b5c4fa3a570f29f704e6dcd (table ksck-test-table [id=16e8c91508bf4659b4f2d3f4531c098d])
I20250115 02:53:24.468050 10837 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:24.475062 10837 catalog_manager.cc:1260] Loaded cluster ID: 468d8bc4dac14121ad866951529e7fdc
I20250115 02:53:24.475322 10837 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:24.480484 10837 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:24.488310 10837 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af: Loaded TSK: 0
I20250115 02:53:24.489307 10837 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:24.510710 10720 heartbeater.cc:501] Master 127.2.36.124:45789 was elected leader, sending a full tablet report...
I20250115 02:53:24.514896 10630 heartbeater.cc:501] Master 127.2.36.124:45789 was elected leader, sending a full tablet report...
I20250115 02:53:24.519057 10552 heartbeater.cc:501] Master 127.2.36.124:45789 was elected leader, sending a full tablet report...
I20250115 02:53:24.529394 10400 catalog_manager.cc:5526] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 reported cstate change: term changed from 1 to 2, leader changed from 2ae6b15bbd244631849f856d84105bdb (127.2.36.66) to fcb481af490f4be18851faed3b37b812 (127.2.36.65). New cstate: current_term: 2 leader_uuid: "fcb481af490f4be18851faed3b37b812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:24.570324 10281 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 3 FOLLOWER]: Refusing update from remote peer 7aeaf33e08c04784847da73e162f05af: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20250115 02:53:24.574666 10831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250115 02:53:24.570308 10346 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 3 FOLLOWER]: Refusing update from remote peer 7aeaf33e08c04784847da73e162f05af: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20250115 02:53:24.578600 10831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250115 02:53:24.620160 10833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aeaf33e08c04784847da73e162f05af. Latest consensus state: current_term: 3 leader_uuid: "7aeaf33e08c04784847da73e162f05af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:24.621052 10833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:24.664273 10831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7aeaf33e08c04784847da73e162f05af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:24.661594 10833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7aeaf33e08c04784847da73e162f05af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:24.666321 10833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:24.666445 10831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:24.696025 10835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7aeaf33e08c04784847da73e162f05af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:24.696787 10835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:24.741748 10858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7aeaf33e08c04784847da73e162f05af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:24.742424 10858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:24.746731 10845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aeaf33e08c04784847da73e162f05af. Latest consensus state: current_term: 3 leader_uuid: "7aeaf33e08c04784847da73e162f05af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:24.747572 10845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:24.752213 10859 raft_consensus.cc:491] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:24.752759 10859 raft_consensus.cc:513] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:24.755551 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fcb481af490f4be18851faed3b37b812" is_pre_election: true
I20250115 02:53:24.770090 10859 leader_election.cc:290] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575), 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:24.771112 10605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb" is_pre_election: true
I20250115 02:53:24.771863 10605 raft_consensus.cc:2463] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f908969f8ee46978fd73c305c92b581 in term 2.
I20250115 02:53:24.775216 10649 leader_election.cc:304] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ae6b15bbd244631849f856d84105bdb, 3f908969f8ee46978fd73c305c92b581; no voters: fcb481af490f4be18851faed3b37b812
I20250115 02:53:24.785190 10859 raft_consensus.cc:2798] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250115 02:53:24.785602 10859 raft_consensus.cc:491] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:24.785949 10859 raft_consensus.cc:3054] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:53:24.796437 10859 raft_consensus.cc:513] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:24.799221 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fcb481af490f4be18851faed3b37b812"
I20250115 02:53:24.801631 10605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de1916181484e67a6929c4b7668904d" candidate_uuid: "3f908969f8ee46978fd73c305c92b581" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2ae6b15bbd244631849f856d84105bdb"
I20250115 02:53:24.802316 10605 raft_consensus.cc:3054] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:53:24.801393 10859 leader_election.cc:290] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 3 election: Requested vote from peers fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575), 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407)
I20250115 02:53:24.810906 10693 raft_consensus.cc:1235] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 3 FOLLOWER]: Rejecting Update request from peer fcb481af490f4be18851faed3b37b812 for earlier term 2. Current term is 3. Ops: []
I20250115 02:53:24.812327 10812 consensus_queue.cc:1046] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250115 02:53:24.813086 10832 raft_consensus.cc:3049] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 2 LEADER]: Stepping down as leader of term 2
I20250115 02:53:24.813421 10832 raft_consensus.cc:738] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fcb481af490f4be18851faed3b37b812, State: Running, Role: LEADER
I20250115 02:53:24.814180 10832 consensus_queue.cc:260] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:24.815413 10832 raft_consensus.cc:3054] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 2 FOLLOWER]: Advancing to term 3
I20250115 02:53:24.821316 10605 raft_consensus.cc:2463] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f908969f8ee46978fd73c305c92b581 in term 3.
I20250115 02:53:24.822113 10604 raft_consensus.cc:1235] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 3 FOLLOWER]: Rejecting Update request from peer fcb481af490f4be18851faed3b37b812 for earlier term 2. Current term is 3. Ops: []
I20250115 02:53:24.833763 10649 leader_election.cc:304] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ae6b15bbd244631849f856d84105bdb, 3f908969f8ee46978fd73c305c92b581; no voters: fcb481af490f4be18851faed3b37b812
I20250115 02:53:24.835268 10859 raft_consensus.cc:2798] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 3 FOLLOWER]: Leader election won for term 3
I20250115 02:53:24.855718 10859 raft_consensus.cc:695] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 3 LEADER]: Becoming Leader. State: Replica: 3f908969f8ee46978fd73c305c92b581, State: Running, Role: LEADER
I20250115 02:53:24.856858 10859 consensus_queue.cc:237] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } }
I20250115 02:53:24.896340 10402 catalog_manager.cc:5526] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 reported cstate change: term changed from 2 to 3, leader changed from fcb481af490f4be18851faed3b37b812 (127.2.36.65) to 3f908969f8ee46978fd73c305c92b581 (127.2.36.67). New cstate: current_term: 3 leader_uuid: "3f908969f8ee46978fd73c305c92b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f908969f8ee46978fd73c305c92b581" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:24.939090 10593 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:53:25.029626 10685 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:53:25.040482 10517 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:53:25.303474 10605 raft_consensus.cc:1270] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 3 FOLLOWER]: Refusing update from remote peer 3f908969f8ee46978fd73c305c92b581: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250115 02:53:25.305052 10859 consensus_queue.cc:1035] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae6b15bbd244631849f856d84105bdb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250115 02:53:25.381847 10526 raft_consensus.cc:1270] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 3 FOLLOWER]: Refusing update from remote peer 3f908969f8ee46978fd73c305c92b581: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch)
I20250115 02:53:25.382256 10526 pending_rounds.cc:77] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812: Aborting all ops after (but not including) 2
I20250115 02:53:25.382477 10526 pending_rounds.cc:99] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812: Aborting uncommitted NO_OP operation due to leader change: 2.3
I20250115 02:53:25.382779 10526 raft_consensus.cc:2883] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250115 02:53:25.384531 10862 consensus_queue.cc:1035] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcb481af490f4be18851faed3b37b812" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 39575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250115 02:53:25.613811 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:53:25.676995 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:25.678036 2193 tablet_replica.cc:331] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812: stopping tablet replica
I20250115 02:53:25.678678 2193 raft_consensus.cc:2238] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:25.679219 2193 raft_consensus.cc:2267] T 6de1916181484e67a6929c4b7668904d P fcb481af490f4be18851faed3b37b812 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:25.681479 2193 tablet_replica.cc:331] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812: stopping tablet replica
I20250115 02:53:25.682226 2193 raft_consensus.cc:2238] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:25.682955 2193 raft_consensus.cc:2267] T dd5854873b5c4fa3a570f29f704e6dcd P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:25.686096 2193 tablet_replica.cc:331] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812: stopping tablet replica
I20250115 02:53:25.686748 2193 raft_consensus.cc:2238] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:25.688995 2193 raft_consensus.cc:2267] T c910d4473d284679b8a2841e01fb7b40 P fcb481af490f4be18851faed3b37b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:25.708731 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:53:25.732981 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
W20250115 02:53:25.757054 10649 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 4 similar messages]
W20250115 02:53:25.775446 10649 consensus_peers.cc:487] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 -> Peer 2ae6b15bbd244631849f856d84105bdb (127.2.36.66:43407): Couldn't send request to peer 2ae6b15bbd244631849f856d84105bdb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250115 02:53:25.799309 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:25.801419 2193 tablet_replica.cc:331] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb: stopping tablet replica
I20250115 02:53:25.803017 2193 raft_consensus.cc:2238] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:25.803922 2193 raft_consensus.cc:2267] T dd5854873b5c4fa3a570f29f704e6dcd P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:25.806118 2193 tablet_replica.cc:331] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb: stopping tablet replica
I20250115 02:53:25.806605 2193 raft_consensus.cc:2238] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:25.807071 2193 raft_consensus.cc:2267] T c910d4473d284679b8a2841e01fb7b40 P 2ae6b15bbd244631849f856d84105bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:25.808825 2193 tablet_replica.cc:331] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb: stopping tablet replica
I20250115 02:53:25.809250 2193 raft_consensus.cc:2238] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 3 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:25.809618 2193 raft_consensus.cc:2267] T 6de1916181484e67a6929c4b7668904d P 2ae6b15bbd244631849f856d84105bdb [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:25.824425 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:53:25.845472 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
W20250115 02:53:25.876616 10650 consensus_peers.cc:487] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 -> Peer fcb481af490f4be18851faed3b37b812 (127.2.36.65:39575): Couldn't send request to peer fcb481af490f4be18851faed3b37b812. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:39575: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:53:25.892481 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:25.893131 2193 tablet_replica.cc:331] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581: stopping tablet replica
I20250115 02:53:25.893693 2193 raft_consensus.cc:2238] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:25.894152 2193 raft_consensus.cc:2267] T dd5854873b5c4fa3a570f29f704e6dcd P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:25.896121 2193 tablet_replica.cc:331] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581: stopping tablet replica
I20250115 02:53:25.896611 2193 raft_consensus.cc:2238] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:25.897061 2193 raft_consensus.cc:2267] T c910d4473d284679b8a2841e01fb7b40 P 3f908969f8ee46978fd73c305c92b581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:25.898731 2193 tablet_replica.cc:331] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581: stopping tablet replica
I20250115 02:53:25.899194 2193 raft_consensus.cc:2238] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 3 LEADER]: Raft consensus shutting down.
I20250115 02:53:25.899945 2193 raft_consensus.cc:2267] T 6de1916181484e67a6929c4b7668904d P 3f908969f8ee46978fd73c305c92b581 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:25.923451 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:53:25.939522 2193 master.cc:537] Master@127.2.36.126:40125 shutting down...
I20250115 02:53:25.967813 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:25.968320 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:25.968622 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 9be9a98525244abfa67175d705720235: stopping tablet replica
I20250115 02:53:25.989988 2193 master.cc:559] Master@127.2.36.126:40125 shutdown complete.
I20250115 02:53:26.009233 2193 master.cc:537] Master@127.2.36.125:40819 shutting down...
I20250115 02:53:26.038194 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 3 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:26.038700 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:26.039049 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 31dc9f4ea6ab494e8a4c3dddb7776b9c: stopping tablet replica
W20250115 02:53:26.067350 10387 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af -> Peer 31dc9f4ea6ab494e8a4c3dddb7776b9c (127.2.36.125:40819): Couldn't send request to peer 31dc9f4ea6ab494e8a4c3dddb7776b9c. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.125:40819: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250115 02:53:26.436689 10389 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af -> Peer 9be9a98525244abfa67175d705720235 (127.2.36.126:40125): Couldn't send request to peer 9be9a98525244abfa67175d705720235. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.126:40125: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:53:26.487198 10877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7aeaf33e08c04784847da73e162f05af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:26.488165 10877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:27.012174 10886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7aeaf33e08c04784847da73e162f05af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be9a98525244abfa67175d705720235" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 40125 } } peers { permanent_uuid: "31dc9f4ea6ab494e8a4c3dddb7776b9c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 40819 } } peers { permanent_uuid: "7aeaf33e08c04784847da73e162f05af" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45789 } } }
I20250115 02:53:27.013074 10886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [sys.catalog]: This master's current role is: LEADER
W20250115 02:53:27.055918 2193 thread.cc:535] Waited for 1000ms trying to join with diag-logger (tid 10368)
W20250115 02:53:28.056370 2193 thread.cc:535] Waited for 2000ms trying to join with diag-logger (tid 10368)
I20250115 02:53:28.132615 2193 master.cc:559] Master@127.2.36.125:40819 shutdown complete.
I20250115 02:53:28.145427 2193 master.cc:537] Master@127.2.36.124:45789 shutting down...
I20250115 02:53:28.165377 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 3 LEADER]: Raft consensus shutting down.
I20250115 02:53:28.166191 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af [term 3 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:28.166494 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7aeaf33e08c04784847da73e162f05af: stopping tablet replica
I20250115 02:53:28.186524 2193 master.cc:559] Master@127.2.36.124:45789 shutdown complete.
[ OK ] RemoteKsckTest.TestMatchingTableFilter (9722 ms)
[ RUN ] RemoteKsckTest.TestFilterOnNotabletTable
I20250115 02:53:28.218330 2193 test_util.cc:274] Using random seed: -936087866
I20250115 02:53:28.219331 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:36515,127.2.36.125:38335,127.2.36.124:38535
I20250115 02:53:28.220590 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:28.226262 10894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:28.227035 10895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:28.227913 10897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.228732 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:28.229729 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:28.229912 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:28.230053 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909608230036 us; error 0 us; skew 500 ppm
I20250115 02:53:28.230520 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:28.239118 2193 webserver.cc:458] Webserver started at http://127.2.36.126:44795/ using document root <none> and password file <none>
I20250115 02:53:28.239578 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:28.239760 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:28.239991 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:28.241010 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "e6d1251df61b4206b288016c970f9cb3"
format_stamp: "Formatted at 2025-01-15 02:53:28 on dist-test-slave-1lv2"
I20250115 02:53:28.245157 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250115 02:53:28.248240 10902 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:28.248957 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250115 02:53:28.249423 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "e6d1251df61b4206b288016c970f9cb3"
format_stamp: "Formatted at 2025-01-15 02:53:28 on dist-test-slave-1lv2"
I20250115 02:53:28.249991 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:28.260630 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:28.261708 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:28.297165 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:36515
I20250115 02:53:28.297258 10953 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:36515 every 8 connection(s)
I20250115 02:53:28.300783 10954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:28.300962 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:28.307046 10956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:28.307868 10957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:28.308746 10959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.308912 10954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:28.310832 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:28.312268 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:28.312539 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:28.312736 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909608312714 us; error 0 us; skew 500 ppm
I20250115 02:53:28.313292 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:28.315856 2193 webserver.cc:458] Webserver started at http://127.2.36.125:42133/ using document root <none> and password file <none>
I20250115 02:53:28.316388 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:28.316625 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:28.316926 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:28.318264 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "9969c8550e5c4f829bc81536e86cff57"
format_stamp: "Formatted at 2025-01-15 02:53:28 on dist-test-slave-1lv2"
I20250115 02:53:28.321486 10954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:28.323361 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
W20250115 02:53:28.326315 10954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:38335: Network error: Client connection negotiation failed: client connection to 127.2.36.125:38335: connect: Connection refused (error 111)
I20250115 02:53:28.326913 10967 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:28.327625 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:53:28.327904 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "9969c8550e5c4f829bc81536e86cff57"
format_stamp: "Formatted at 2025-01-15 02:53:28 on dist-test-slave-1lv2"
I20250115 02:53:28.328159 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:28.352324 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:28.353433 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:28.356856 10954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } attempt: 1
W20250115 02:53:28.360838 10954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:38335: Network error: Client connection negotiation failed: client connection to 127.2.36.125:38335: connect: Connection refused (error 111)
I20250115 02:53:28.389963 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:38335
I20250115 02:53:28.390064 11018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:38335 every 8 connection(s)
I20250115 02:53:28.393554 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:28.393658 11019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:28.395327 10954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } attempt: 2
W20250115 02:53:28.401655 11021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.401899 11019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:28.402655 11023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.412842 10954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:28.413506 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:28.414263 11026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.415287 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:28.415593 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:28.415817 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909608415788 us; error 0 us; skew 500 ppm
I20250115 02:53:28.416378 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250115 02:53:28.418246 10954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:38535: Network error: Client connection negotiation failed: client connection to 127.2.36.124:38535: connect: Connection refused (error 111)
I20250115 02:53:28.420651 2193 webserver.cc:458] Webserver started at http://127.2.36.124:46081/ using document root <none> and password file <none>
I20250115 02:53:28.421386 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:28.421619 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:28.421873 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:28.422400 11019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:28.423039 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "8237eef53aa04c5296dcfcec9a7c3382"
format_stamp: "Formatted at 2025-01-15 02:53:28 on dist-test-slave-1lv2"
I20250115 02:53:28.428095 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250115 02:53:28.431808 11019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:28.431926 11031 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:28.432771 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:53:28.433043 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "8237eef53aa04c5296dcfcec9a7c3382"
format_stamp: "Formatted at 2025-01-15 02:53:28 on dist-test-slave-1lv2"
I20250115 02:53:28.433363 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250115 02:53:28.437044 11019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:38535: Network error: Client connection negotiation failed: client connection to 127.2.36.124:38535: connect: Connection refused (error 111)
I20250115 02:53:28.447711 10954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } attempt: 1
I20250115 02:53:28.449879 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:28.451143 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250115 02:53:28.451673 10954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:38535: Network error: Client connection negotiation failed: client connection to 127.2.36.124:38535: connect: Connection refused (error 111)
I20250115 02:53:28.476580 11019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } attempt: 1
W20250115 02:53:28.480492 11019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:38535: Network error: Client connection negotiation failed: client connection to 127.2.36.124:38535: connect: Connection refused (error 111)
I20250115 02:53:28.484159 10954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } attempt: 2
W20250115 02:53:28.488691 10954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:38535: Network error: Client connection negotiation failed: client connection to 127.2.36.124:38535: connect: Connection refused (error 111)
I20250115 02:53:28.489457 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:38535
I20250115 02:53:28.489564 11083 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:38535 every 8 connection(s)
I20250115 02:53:28.492547 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:53:28.493468 11084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:28.499465 11084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:28.509217 11084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:28.517288 11084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:28.532107 11084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382: Bootstrap starting.
I20250115 02:53:28.532963 11019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } attempt: 2
I20250115 02:53:28.538014 11084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:28.543216 11084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382: No bootstrap required, opened a new log
I20250115 02:53:28.545315 11084 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } }
I20250115 02:53:28.545871 11084 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:28.546093 11084 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8237eef53aa04c5296dcfcec9a7c3382, State: Initialized, Role: FOLLOWER
I20250115 02:53:28.546670 11084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } }
I20250115 02:53:28.548522 11019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57: Bootstrap starting.
I20250115 02:53:28.548493 11089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } } }
I20250115 02:53:28.549427 11089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:28.549769 11084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:28.554641 11019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:28.559514 11019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57: No bootstrap required, opened a new log
I20250115 02:53:28.561303 10954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } attempt: 3
I20250115 02:53:28.561811 11019 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } }
I20250115 02:53:28.562460 11019 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:28.562724 11019 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9969c8550e5c4f829bc81536e86cff57, State: Initialized, Role: FOLLOWER
I20250115 02:53:28.563354 11019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } }
I20250115 02:53:28.565300 11102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } } }
W20250115 02:53:28.566112 11101 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250115 02:53:28.566135 11102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [sys.catalog]: This master's current role is: FOLLOWER
W20250115 02:53:28.566622 11101 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:28.567134 11019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:28.577801 10954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3: Bootstrap starting.
W20250115 02:53:28.583381 11113 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:28.583768 11113 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:28.584143 10954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:28.588083 10954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3: No bootstrap required, opened a new log
I20250115 02:53:28.589965 10954 raft_consensus.cc:357] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } }
I20250115 02:53:28.590478 10954 raft_consensus.cc:383] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:28.590655 10954 raft_consensus.cc:738] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d1251df61b4206b288016c970f9cb3, State: Initialized, Role: FOLLOWER
I20250115 02:53:28.591136 10954 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } }
I20250115 02:53:28.592625 11115 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } } }
I20250115 02:53:28.593209 11115 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:28.593881 10954 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:28.602519 11115 raft_consensus.cc:491] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:28.603065 11115 raft_consensus.cc:513] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } }
I20250115 02:53:28.604710 11115 leader_election.cc:290] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9969c8550e5c4f829bc81536e86cff57 (127.2.36.125:38335), 8237eef53aa04c5296dcfcec9a7c3382 (127.2.36.124:38535)
I20250115 02:53:28.605536 10994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6d1251df61b4206b288016c970f9cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9969c8550e5c4f829bc81536e86cff57" is_pre_election: true
I20250115 02:53:28.605794 11059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6d1251df61b4206b288016c970f9cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8237eef53aa04c5296dcfcec9a7c3382" is_pre_election: true
I20250115 02:53:28.606062 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:53:28.606418 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:53:28.606434 11059 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6d1251df61b4206b288016c970f9cb3 in term 0.
I20250115 02:53:28.606278 10994 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6d1251df61b4206b288016c970f9cb3 in term 0.
I20250115 02:53:28.607988 10905 leader_election.cc:304] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8237eef53aa04c5296dcfcec9a7c3382, e6d1251df61b4206b288016c970f9cb3; no voters:
I20250115 02:53:28.608369 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:28.608924 11115 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:28.609280 11115 raft_consensus.cc:491] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:28.609601 11115 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 0 FOLLOWER]: Advancing to term 1
W20250115 02:53:28.611514 11126 catalog_manager.cc:1559] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:28.611876 11126 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:53:28.615814 11127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.616492 11115 raft_consensus.cc:513] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } }
W20250115 02:53:28.619290 11128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.620129 10994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6d1251df61b4206b288016c970f9cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9969c8550e5c4f829bc81536e86cff57"
I20250115 02:53:28.620859 10994 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:28.622012 11059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6d1251df61b4206b288016c970f9cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8237eef53aa04c5296dcfcec9a7c3382"
I20250115 02:53:28.622637 11059 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:28.623734 11115 leader_election.cc:290] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [CANDIDATE]: Term 1 election: Requested vote from peers 9969c8550e5c4f829bc81536e86cff57 (127.2.36.125:38335), 8237eef53aa04c5296dcfcec9a7c3382 (127.2.36.124:38535)
W20250115 02:53:28.625190 11130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.625931 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:28.626833 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:28.627034 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:28.627220 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909608627194 us; error 0 us; skew 500 ppm
I20250115 02:53:28.627730 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:28.628484 10994 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6d1251df61b4206b288016c970f9cb3 in term 1.
I20250115 02:53:28.629566 10904 leader_election.cc:304] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9969c8550e5c4f829bc81536e86cff57, e6d1251df61b4206b288016c970f9cb3; no voters:
I20250115 02:53:28.629940 11059 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6d1251df61b4206b288016c970f9cb3 in term 1.
I20250115 02:53:28.630410 11115 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:28.630903 2193 webserver.cc:458] Webserver started at http://127.2.36.65:41389/ using document root <none> and password file <none>
I20250115 02:53:28.631376 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:28.631548 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:28.631793 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:28.631933 11115 raft_consensus.cc:695] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 1 LEADER]: Becoming Leader. State: Replica: e6d1251df61b4206b288016c970f9cb3, State: Running, Role: LEADER
I20250115 02:53:28.632807 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "0c176e0eb85046bb98a030d22b3ae250"
format_stamp: "Formatted at 2025-01-15 02:53:28 on dist-test-slave-1lv2"
I20250115 02:53:28.632700 11115 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } }
I20250115 02:53:28.637245 11134 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6d1251df61b4206b288016c970f9cb3. Latest consensus state: current_term: 1 leader_uuid: "e6d1251df61b4206b288016c970f9cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } } }
I20250115 02:53:28.637889 11134 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:28.637945 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250115 02:53:28.639442 11137 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:28.641992 11138 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:28.642840 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250115 02:53:28.643106 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "0c176e0eb85046bb98a030d22b3ae250"
format_stamp: "Formatted at 2025-01-15 02:53:28 on dist-test-slave-1lv2"
I20250115 02:53:28.643347 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:28.644389 11137 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:28.653422 10994 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [term 1 FOLLOWER]: Refusing update from remote peer e6d1251df61b4206b288016c970f9cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:28.653589 11059 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [term 1 FOLLOWER]: Refusing update from remote peer e6d1251df61b4206b288016c970f9cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:28.654708 11115 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:28.655388 11134 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:28.667485 11089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6d1251df61b4206b288016c970f9cb3. Latest consensus state: current_term: 1 leader_uuid: "e6d1251df61b4206b288016c970f9cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } } }
I20250115 02:53:28.668275 11089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:28.671921 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:28.673640 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:28.676214 11115 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6d1251df61b4206b288016c970f9cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } } }
I20250115 02:53:28.677096 11115 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:28.678388 11137 catalog_manager.cc:1348] Generated new cluster ID: ada73492d09d41cc8bb0255f42d00e0d
I20250115 02:53:28.678791 11137 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:28.679180 11102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6d1251df61b4206b288016c970f9cb3. Latest consensus state: current_term: 1 leader_uuid: "e6d1251df61b4206b288016c970f9cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } } }
I20250115 02:53:28.679956 11102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:28.683238 11134 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6d1251df61b4206b288016c970f9cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } } }
I20250115 02:53:28.683982 11134 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:28.685724 11089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6d1251df61b4206b288016c970f9cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } } }
I20250115 02:53:28.686444 11089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:28.690547 11102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6d1251df61b4206b288016c970f9cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } } }
I20250115 02:53:28.691329 11102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:28.694109 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:28.697243 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:28.697517 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:28.697830 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:28.698033 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:28.703454 11137 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:53:28.705101 11137 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:28.738479 11137 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3: Generated new TSK 0
I20250115 02:53:28.739300 11137 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:28.759771 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:43759
I20250115 02:53:28.759871 11209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:43759 every 8 connection(s)
I20250115 02:53:28.767343 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:28.788744 11210 heartbeater.cc:346] Connected to a master server at 127.2.36.124:38535
I20250115 02:53:28.789146 11210 heartbeater.cc:463] Registering TS with master...
W20250115 02:53:28.789775 11217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.790032 11210 heartbeater.cc:509] Master 127.2.36.124:38535 requested a full tablet report, sending...
I20250115 02:53:28.792443 11211 heartbeater.cc:346] Connected to a master server at 127.2.36.125:38335
I20250115 02:53:28.792795 11211 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:28.793280 11049 ts_manager.cc:194] Registered new tserver with Master: 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759)
I20250115 02:53:28.793596 11211 heartbeater.cc:509] Master 127.2.36.125:38335 requested a full tablet report, sending...
I20250115 02:53:28.796986 10984 ts_manager.cc:194] Registered new tserver with Master: 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759)
W20250115 02:53:28.798220 11218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.800604 11212 heartbeater.cc:346] Connected to a master server at 127.2.36.126:36515
W20250115 02:53:28.800854 11220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.801017 11212 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:28.801774 11212 heartbeater.cc:509] Master 127.2.36.126:36515 requested a full tablet report, sending...
I20250115 02:53:28.801828 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:28.802671 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:28.802925 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:28.803093 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909608803076 us; error 0 us; skew 500 ppm
I20250115 02:53:28.803575 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:28.803728 10919 ts_manager.cc:194] Registered new tserver with Master: 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759)
I20250115 02:53:28.805939 10919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47952
I20250115 02:53:28.806206 2193 webserver.cc:458] Webserver started at http://127.2.36.66:43947/ using document root <none> and password file <none>
I20250115 02:53:28.806718 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:28.806943 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:28.807191 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:28.808215 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "a17d3d0a6e924761b7beb3110bf2481d"
format_stamp: "Formatted at 2025-01-15 02:53:28 on dist-test-slave-1lv2"
I20250115 02:53:28.812819 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20250115 02:53:28.815920 11225 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:28.816634 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:53:28.816929 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "a17d3d0a6e924761b7beb3110bf2481d"
format_stamp: "Formatted at 2025-01-15 02:53:28 on dist-test-slave-1lv2"
I20250115 02:53:28.817199 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:28.819036 11212 heartbeater.cc:501] Master 127.2.36.126:36515 was elected leader, sending a full tablet report...
I20250115 02:53:28.843739 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:28.845031 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:28.846241 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:28.850150 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:28.850369 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:28.850646 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:28.850878 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250115 02:53:28.898020 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:43955
I20250115 02:53:28.898137 11287 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:43955 every 8 connection(s)
I20250115 02:53:28.911020 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:28.925388 11289 heartbeater.cc:346] Connected to a master server at 127.2.36.125:38335
I20250115 02:53:28.925827 11289 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:28.926666 11289 heartbeater.cc:509] Master 127.2.36.125:38335 requested a full tablet report, sending...
I20250115 02:53:28.929414 10984 ts_manager.cc:194] Registered new tserver with Master: a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955)
W20250115 02:53:28.935936 11295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.939607 11288 heartbeater.cc:346] Connected to a master server at 127.2.36.124:38535
I20250115 02:53:28.940014 11288 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:28.940897 11288 heartbeater.cc:509] Master 127.2.36.124:38535 requested a full tablet report, sending...
I20250115 02:53:28.943718 11049 ts_manager.cc:194] Registered new tserver with Master: a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955)
I20250115 02:53:28.945222 11290 heartbeater.cc:346] Connected to a master server at 127.2.36.126:36515
I20250115 02:53:28.945534 11290 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:28.946533 11290 heartbeater.cc:509] Master 127.2.36.126:36515 requested a full tablet report, sending...
W20250115 02:53:28.947312 11296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.948082 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:28.948699 10919 ts_manager.cc:194] Registered new tserver with Master: a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955)
W20250115 02:53:28.949649 11298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:28.950701 10919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47966
I20250115 02:53:28.950834 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:28.951146 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:28.951349 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909608951331 us; error 0 us; skew 500 ppm
I20250115 02:53:28.951988 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:28.954710 2193 webserver.cc:458] Webserver started at http://127.2.36.67:41491/ using document root <none> and password file <none>
I20250115 02:53:28.955258 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:28.955495 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:28.955804 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:28.957197 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "cb6e4bf49c05498983e41c6e1da2c9da"
format_stamp: "Formatted at 2025-01-15 02:53:28 on dist-test-slave-1lv2"
I20250115 02:53:28.962026 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:53:28.964035 11290 heartbeater.cc:501] Master 127.2.36.126:36515 was elected leader, sending a full tablet report...
I20250115 02:53:28.967190 11303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:28.968217 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250115 02:53:28.968546 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "cb6e4bf49c05498983e41c6e1da2c9da"
format_stamp: "Formatted at 2025-01-15 02:53:28 on dist-test-slave-1lv2"
I20250115 02:53:28.968844 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:28.995038 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:28.996402 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:28.997690 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:28.999768 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:29.000027 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:29.000289 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:29.000480 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:29.061970 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:33107
I20250115 02:53:29.062124 11365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:33107 every 8 connection(s)
I20250115 02:53:29.091560 11366 heartbeater.cc:346] Connected to a master server at 127.2.36.124:38535
I20250115 02:53:29.092103 11366 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:29.093034 11366 heartbeater.cc:509] Master 127.2.36.124:38535 requested a full tablet report, sending...
I20250115 02:53:29.097590 11049 ts_manager.cc:194] Registered new tserver with Master: cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
I20250115 02:53:29.109318 11369 heartbeater.cc:346] Connected to a master server at 127.2.36.126:36515
I20250115 02:53:29.109688 11369 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:29.109721 11367 heartbeater.cc:346] Connected to a master server at 127.2.36.125:38335
I20250115 02:53:29.110241 11367 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:29.110615 11369 heartbeater.cc:509] Master 127.2.36.126:36515 requested a full tablet report, sending...
I20250115 02:53:29.110975 11367 heartbeater.cc:509] Master 127.2.36.125:38335 requested a full tablet report, sending...
I20250115 02:53:29.113108 10918 ts_manager.cc:194] Registered new tserver with Master: cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
I20250115 02:53:29.113628 10984 ts_manager.cc:194] Registered new tserver with Master: cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
I20250115 02:53:29.114674 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006543479s
I20250115 02:53:29.115005 10918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47968
I20250115 02:53:29.130455 11369 heartbeater.cc:501] Master 127.2.36.126:36515 was elected leader, sending a full tablet report...
I20250115 02:53:29.165827 10919 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47984:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:53:29.168639 10919 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:53:29.284080 11175 tablet_service.cc:1467] Processing CreateTablet for tablet 17cf8b9076fe465a99b51a0193993e76 (DEFAULT_TABLE table=ksck-test-table [id=5f1f32346cbe4a209a7560e6bd395c07]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:29.285470 11175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17cf8b9076fe465a99b51a0193993e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:29.287652 11174 tablet_service.cc:1467] Processing CreateTablet for tablet f8c59a2688344aa394e250fecdcafa33 (DEFAULT_TABLE table=ksck-test-table [id=5f1f32346cbe4a209a7560e6bd395c07]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:29.288988 11174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c59a2688344aa394e250fecdcafa33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:29.291712 11173 tablet_service.cc:1467] Processing CreateTablet for tablet f5f3c76310ef4f0c84c12e6fd44fbe3b (DEFAULT_TABLE table=ksck-test-table [id=5f1f32346cbe4a209a7560e6bd395c07]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:29.292349 11331 tablet_service.cc:1467] Processing CreateTablet for tablet 17cf8b9076fe465a99b51a0193993e76 (DEFAULT_TABLE table=ksck-test-table [id=5f1f32346cbe4a209a7560e6bd395c07]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:29.292976 11173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f3c76310ef4f0c84c12e6fd44fbe3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:29.293821 11329 tablet_service.cc:1467] Processing CreateTablet for tablet f5f3c76310ef4f0c84c12e6fd44fbe3b (DEFAULT_TABLE table=ksck-test-table [id=5f1f32346cbe4a209a7560e6bd395c07]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:29.293742 11331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17cf8b9076fe465a99b51a0193993e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:29.292445 11330 tablet_service.cc:1467] Processing CreateTablet for tablet f8c59a2688344aa394e250fecdcafa33 (DEFAULT_TABLE table=ksck-test-table [id=5f1f32346cbe4a209a7560e6bd395c07]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:29.296921 11330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c59a2688344aa394e250fecdcafa33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:29.297924 11329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f3c76310ef4f0c84c12e6fd44fbe3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:29.298492 11253 tablet_service.cc:1467] Processing CreateTablet for tablet 17cf8b9076fe465a99b51a0193993e76 (DEFAULT_TABLE table=ksck-test-table [id=5f1f32346cbe4a209a7560e6bd395c07]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:29.299793 11253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17cf8b9076fe465a99b51a0193993e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:29.303694 11252 tablet_service.cc:1467] Processing CreateTablet for tablet f8c59a2688344aa394e250fecdcafa33 (DEFAULT_TABLE table=ksck-test-table [id=5f1f32346cbe4a209a7560e6bd395c07]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:29.305015 11252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c59a2688344aa394e250fecdcafa33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:29.306674 11251 tablet_service.cc:1467] Processing CreateTablet for tablet f5f3c76310ef4f0c84c12e6fd44fbe3b (DEFAULT_TABLE table=ksck-test-table [id=5f1f32346cbe4a209a7560e6bd395c07]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:29.307916 11251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f3c76310ef4f0c84c12e6fd44fbe3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:29.409657 11388 tablet_bootstrap.cc:492] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250: Bootstrap starting.
I20250115 02:53:29.423353 11388 tablet_bootstrap.cc:654] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:29.435142 11390 tablet_bootstrap.cc:492] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da: Bootstrap starting.
I20250115 02:53:29.435783 11389 tablet_bootstrap.cc:492] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d: Bootstrap starting.
I20250115 02:53:29.441809 11388 tablet_bootstrap.cc:492] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250: No bootstrap required, opened a new log
I20250115 02:53:29.442576 11388 ts_tablet_manager.cc:1397] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.000s
I20250115 02:53:29.446549 11388 raft_consensus.cc:357] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:29.447513 11388 raft_consensus.cc:383] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:29.447878 11388 raft_consensus.cc:738] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c176e0eb85046bb98a030d22b3ae250, State: Initialized, Role: FOLLOWER
I20250115 02:53:29.448801 11388 consensus_queue.cc:260] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:29.452461 11390 tablet_bootstrap.cc:654] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:29.559934 11394 raft_consensus.cc:491] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:29.566244 11390 tablet_bootstrap.cc:492] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da: No bootstrap required, opened a new log
I20250115 02:53:29.625893 11394 raft_consensus.cc:513] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:29.627242 11113 catalog_manager.cc:1260] Loaded cluster ID: ada73492d09d41cc8bb0255f42d00e0d
I20250115 02:53:29.627667 11113 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57: loading cluster ID for follower catalog manager: success
I20250115 02:53:29.631011 11390 ts_tablet_manager.cc:1397] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da: Time spent bootstrapping tablet: real 0.196s user 0.046s sys 0.039s
I20250115 02:53:29.632711 11101 catalog_manager.cc:1260] Loaded cluster ID: ada73492d09d41cc8bb0255f42d00e0d
I20250115 02:53:29.634963 11390 raft_consensus.cc:357] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:29.640148 11390 raft_consensus.cc:383] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:29.640450 11390 raft_consensus.cc:738] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e4bf49c05498983e41c6e1da2c9da, State: Initialized, Role: FOLLOWER
I20250115 02:53:29.641278 11390 consensus_queue.cc:260] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:29.660974 11389 tablet_bootstrap.cc:654] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:29.663013 11394 leader_election.cc:290] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955), cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
I20250115 02:53:29.665258 11388 ts_tablet_manager.cc:1428] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250: Time spent starting tablet: real 0.222s user 0.071s sys 0.097s
I20250115 02:53:29.693042 11390 ts_tablet_manager.cc:1428] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da: Time spent starting tablet: real 0.062s user 0.004s sys 0.018s
I20250115 02:53:29.694689 11390 tablet_bootstrap.cc:492] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da: Bootstrap starting.
I20250115 02:53:29.701402 11390 tablet_bootstrap.cc:654] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:29.706349 11390 tablet_bootstrap.cc:492] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da: No bootstrap required, opened a new log
I20250115 02:53:29.706933 11390 ts_tablet_manager.cc:1397] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.005s
I20250115 02:53:29.709815 11390 raft_consensus.cc:357] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:29.710631 11390 raft_consensus.cc:383] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:29.710986 11390 raft_consensus.cc:738] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e4bf49c05498983e41c6e1da2c9da, State: Initialized, Role: FOLLOWER
I20250115 02:53:29.711647 11390 consensus_queue.cc:260] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:29.713692 11390 ts_tablet_manager.cc:1428] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20250115 02:53:29.714612 11390 tablet_bootstrap.cc:492] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da: Bootstrap starting.
I20250115 02:53:29.719660 11390 tablet_bootstrap.cc:654] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:29.724186 11390 tablet_bootstrap.cc:492] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da: No bootstrap required, opened a new log
I20250115 02:53:29.724582 11390 ts_tablet_manager.cc:1397] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20250115 02:53:29.766917 11390 raft_consensus.cc:357] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:29.769220 11390 raft_consensus.cc:383] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:29.769604 11390 raft_consensus.cc:738] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e4bf49c05498983e41c6e1da2c9da, State: Initialized, Role: FOLLOWER
I20250115 02:53:29.770726 11390 consensus_queue.cc:260] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:30.428464 11390 ts_tablet_manager.cc:1428] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da: Time spent starting tablet: real 0.704s user 0.005s sys 0.011s
W20250115 02:53:30.429018 11390 ts_tablet_manager.cc:1460] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da: Tablet startup took 704ms
W20250115 02:53:30.430016 11390 ts_tablet_manager.cc:1462] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da: Trace:
0115 02:53:29.291141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:29.291748 (+ 607us) service_pool.cc:225] Handling call
0115 02:53:29.296405 (+ 4657us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0115 02:53:29.296510 (+ 105us) ts_tablet_manager.cc:676] Creating new metadata...
0115 02:53:29.363484 (+ 66974us) tablet_metadata.cc:641] Metadata flushed
0115 02:53:29.429365 (+ 65881us) tablet_replica.cc:206] Creating consensus instance
0115 02:53:29.429901 (+ 536us) inbound_call.cc:173] Queueing success response
0115 02:53:29.714330 (+284429us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0115 02:53:29.724798 (+ 10468us) ts_tablet_manager.cc:1429] Starting tablet replica
0115 02:53:29.761677 (+ 36879us) tablet_replica.cc:260] Starting instrumentation
0115 02:53:29.762994 (+ 1317us) tablet_replica.cc:279] Starting consensus
Metrics: {"raft.queue_time_us":1238,"raft.run_cpu_time_us":655,"raft.run_wall_time_us":652,"tablet-open.queue_time_us":284512,"thread_start_us":726,"threads_started":1}
W20250115 02:53:30.524119 11142 thread.cc:635] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.886s user 0.009s sys 0.013s
W20250115 02:53:30.524598 11142 thread.cc:601] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.887s user 0.009s sys 0.014s
I20250115 02:53:29.635881 11101 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382: loading cluster ID for follower catalog manager: success
I20250115 02:53:30.543104 11101 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382: acquiring CA information for follower catalog manager: success
I20250115 02:53:30.547578 11341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f3c76310ef4f0c84c12e6fd44fbe3b" candidate_uuid: "0c176e0eb85046bb98a030d22b3ae250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" is_pre_election: true
I20250115 02:53:30.548478 11101 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:30.548967 11341 raft_consensus.cc:2463] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c176e0eb85046bb98a030d22b3ae250 in term 0.
I20250115 02:53:30.550647 11142 leader_election.cc:304] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c176e0eb85046bb98a030d22b3ae250, cb6e4bf49c05498983e41c6e1da2c9da; no voters:
I20250115 02:53:30.555914 11403 raft_consensus.cc:2798] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:30.556408 11403 raft_consensus.cc:491] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:30.556810 11403 raft_consensus.cc:3054] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:30.558445 11389 tablet_bootstrap.cc:492] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d: No bootstrap required, opened a new log
I20250115 02:53:30.559444 11389 ts_tablet_manager.cc:1397] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d: Time spent bootstrapping tablet: real 1.124s user 0.021s sys 0.017s
I20250115 02:53:30.563705 11389 raft_consensus.cc:357] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:30.564704 11389 raft_consensus.cc:383] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:30.565037 11389 raft_consensus.cc:738] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17d3d0a6e924761b7beb3110bf2481d, State: Initialized, Role: FOLLOWER
I20250115 02:53:30.566082 11389 consensus_queue.cc:260] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:30.568504 11402 raft_consensus.cc:491] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:30.568961 11403 raft_consensus.cc:513] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:30.569415 11402 raft_consensus.cc:513] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:30.572703 11403 leader_election.cc:290] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [CANDIDATE]: Term 1 election: Requested vote from peers a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955), cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
I20250115 02:53:30.572989 11341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f3c76310ef4f0c84c12e6fd44fbe3b" candidate_uuid: "0c176e0eb85046bb98a030d22b3ae250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6e4bf49c05498983e41c6e1da2c9da"
I20250115 02:53:30.573625 11341 raft_consensus.cc:3054] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:30.576078 11402 leader_election.cc:290] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955)
I20250115 02:53:30.583535 11341 raft_consensus.cc:2463] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c176e0eb85046bb98a030d22b3ae250 in term 1.
I20250115 02:53:30.587002 11113 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57: acquiring CA information for follower catalog manager: success
I20250115 02:53:30.588958 11142 leader_election.cc:304] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c176e0eb85046bb98a030d22b3ae250, cb6e4bf49c05498983e41c6e1da2c9da; no voters:
I20250115 02:53:30.590019 11403 raft_consensus.cc:2798] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:30.590423 11389 ts_tablet_manager.cc:1428] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d: Time spent starting tablet: real 0.031s user 0.009s sys 0.004s
I20250115 02:53:30.592245 11389 tablet_bootstrap.cc:492] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d: Bootstrap starting.
I20250115 02:53:30.596666 11113 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:30.598985 10983 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58304 (request call id 16) took 1219 ms. Trace:
I20250115 02:53:30.599412 10983 rpcz_store.cc:276] 0115 02:53:29.379659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:29.379863 (+ 204us) service_pool.cc:225] Handling call
0115 02:53:29.393378 (+ 13515us) spinlock_profiling.cc:243] Waited 2.49 ms on lock 0x7b5400006f90. stack: 00007f5a18a69f63 00007f5a18a69e60 00007f5a183bb584 00007f5a2242cff4 00007f5a2242cf89 00007f5a188df911 00007f5a18a5e991 00007f5a18a5eed2 00007f5a21535779 00007f5a214cde06 00007f5a2149eca8 00007f5a21596dee 00007f5a1fbfefc4 00007f5a1fbfef52 00007f5a1fbfee81 00007f5a1fbfedfc
0115 02:53:30.597955 (+1204577us) spinlock_profiling.cc:243] Waited 7.23 ms on lock 0x7b5400567a54. stack: 00007f5a18a69f63 00007f5a18a69e60 00007f5a183bb584 00007f5a2242cff4 00007f5a2242cf89 00007f5a1b595d50 00007f5a1b618f2a 00007f5a2149edec 00007f5a21596dee 00007f5a1fbfefc4 00007f5a1fbfef52 00007f5a1fbfee81 00007f5a1fbfedfc 00007f5a1fbfe0b2 00007f5a198d3dfc 00007f5a198d3236
0115 02:53:30.598820 (+ 865us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22343936}
I20250115 02:53:30.600347 11389 tablet_bootstrap.cc:654] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:30.605072 11403 raft_consensus.cc:695] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 1 LEADER]: Becoming Leader. State: Replica: 0c176e0eb85046bb98a030d22b3ae250, State: Running, Role: LEADER
I20250115 02:53:30.606386 11403 consensus_queue.cc:237] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
W20250115 02:53:30.607573 11141 thread.cc:635] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.978s user 0.018s sys 0.025s
W20250115 02:53:30.607952 11141 thread.cc:601] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.979s user 0.018s sys 0.025s
I20250115 02:53:30.612609 11389 tablet_bootstrap.cc:492] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d: No bootstrap required, opened a new log
I20250115 02:53:30.613131 11389 ts_tablet_manager.cc:1397] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.000s
I20250115 02:53:30.616227 11389 raft_consensus.cc:357] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:30.617122 11389 raft_consensus.cc:383] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:30.617441 11389 raft_consensus.cc:738] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17d3d0a6e924761b7beb3110bf2481d, State: Initialized, Role: FOLLOWER
I20250115 02:53:30.617369 11263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f3c76310ef4f0c84c12e6fd44fbe3b" candidate_uuid: "0c176e0eb85046bb98a030d22b3ae250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17d3d0a6e924761b7beb3110bf2481d" is_pre_election: true
I20250115 02:53:30.618700 11389 consensus_queue.cc:260] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:30.621099 11389 ts_tablet_manager.cc:1428] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20250115 02:53:30.621909 11141 leader_election.cc:343] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:30.623847 11389 tablet_bootstrap.cc:492] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d: Bootstrap starting.
I20250115 02:53:30.617660 11262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f3c76310ef4f0c84c12e6fd44fbe3b" candidate_uuid: "0c176e0eb85046bb98a030d22b3ae250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17d3d0a6e924761b7beb3110bf2481d"
W20250115 02:53:30.625847 11141 leader_election.cc:343] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:30.628624 10916 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47966 (request call id 28) took 1050 ms. Trace:
I20250115 02:53:30.628968 10916 rpcz_store.cc:276] 0115 02:53:29.578155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:29.578357 (+ 202us) service_pool.cc:225] Handling call
0115 02:53:30.627455 (+1049098us) spinlock_profiling.cc:243] Waited 40.6 ms on lock 0x7b540009dc10. stack: 00007f5a18a69f63 00007f5a18a69e60 00007f5a183bb584 00007f5a2242cff4 00007f5a2242cf89 00007f5a188df911 00007f5a18a5e991 00007f5a18a5eed2 00007f5a21535779 00007f5a214cde06 00007f5a2149eca8 00007f5a21596dee 00007f5a1fbfefc4 00007f5a1fbfef52 00007f5a1fbfee81 00007f5a1fbfedfc
0115 02:53:30.628471 (+ 1016us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":0,"spinlock_wait_cycles":93270784}
I20250115 02:53:30.631455 11389 tablet_bootstrap.cc:654] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:30.634028 11047 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35086 (request call id 33) took 1180 ms. Trace:
I20250115 02:53:30.637990 11388 tablet_bootstrap.cc:492] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250: Bootstrap starting.
I20250115 02:53:30.639320 11262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cf8b9076fe465a99b51a0193993e76" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17d3d0a6e924761b7beb3110bf2481d" is_pre_election: true
I20250115 02:53:30.640110 11262 raft_consensus.cc:2463] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6e4bf49c05498983e41c6e1da2c9da in term 0.
I20250115 02:53:30.634482 11047 rpcz_store.cc:276] 0115 02:53:29.453875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:29.454090 (+ 215us) service_pool.cc:225] Handling call
0115 02:53:30.632585 (+1178495us) spinlock_profiling.cc:243] Waited 988 us on lock 0x7b540051e410. stack: 00007f5a18a69f63 00007f5a18a69e60 00007f5a183bb584 00007f5a2242cff4 00007f5a2242cf89 00007f5a188df911 00007f5a18a5e991 00007f5a18a5eed2 00007f5a21535779 00007f5a214cde06 00007f5a2149eca8 00007f5a21596dee 00007f5a1fbfefc4 00007f5a1fbfef52 00007f5a1fbfee81 00007f5a1fbfedfc
0115 02:53:30.633827 (+ 1242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2271488}
I20250115 02:53:30.641609 11306 leader_election.cc:304] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a17d3d0a6e924761b7beb3110bf2481d, cb6e4bf49c05498983e41c6e1da2c9da; no voters:
I20250115 02:53:30.642544 11402 raft_consensus.cc:2798] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:30.642917 11402 raft_consensus.cc:491] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:30.643246 11402 raft_consensus.cc:3054] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:30.656100 11388 tablet_bootstrap.cc:654] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:30.657999 10919 catalog_manager.cc:5526] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "0c176e0eb85046bb98a030d22b3ae250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:30.682996 11402 raft_consensus.cc:513] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:30.685662 11402 leader_election.cc:290] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 election: Requested vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955)
W20250115 02:53:30.687732 10950 debug-util.cc:398] Leaking SignalData structure 0x7b08001b5a80 after lost signal to thread 2196
W20250115 02:53:30.689090 10950 debug-util.cc:398] Leaking SignalData structure 0x7b08000a4760 after lost signal to thread 10953
W20250115 02:53:30.690156 10950 debug-util.cc:398] Leaking SignalData structure 0x7b08001ce380 after lost signal to thread 11018
I20250115 02:53:30.690364 11262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cf8b9076fe465a99b51a0193993e76" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17d3d0a6e924761b7beb3110bf2481d"
I20250115 02:53:30.691007 11262 raft_consensus.cc:3054] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Advancing to term 1
W20250115 02:53:30.694453 10950 debug-util.cc:398] Leaking SignalData structure 0x7b08001925e0 after lost signal to thread 11083
W20250115 02:53:30.698266 10950 debug-util.cc:398] Leaking SignalData structure 0x7b08001b49c0 after lost signal to thread 11365
I20250115 02:53:30.708539 11262 raft_consensus.cc:2463] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6e4bf49c05498983e41c6e1da2c9da in term 1.
I20250115 02:53:30.716512 11402 raft_consensus.cc:491] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:30.717111 11402 raft_consensus.cc:513] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:30.719287 11402 leader_election.cc:290] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955)
I20250115 02:53:30.722446 11263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c59a2688344aa394e250fecdcafa33" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17d3d0a6e924761b7beb3110bf2481d" is_pre_election: true
I20250115 02:53:30.723181 11263 raft_consensus.cc:2463] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6e4bf49c05498983e41c6e1da2c9da in term 0.
I20250115 02:53:30.730382 11306 leader_election.cc:304] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a17d3d0a6e924761b7beb3110bf2481d, cb6e4bf49c05498983e41c6e1da2c9da; no voters:
I20250115 02:53:30.736959 11389 tablet_bootstrap.cc:492] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d: No bootstrap required, opened a new log
I20250115 02:53:30.737692 11389 ts_tablet_manager.cc:1397] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d: Time spent bootstrapping tablet: real 0.114s user 0.037s sys 0.048s
I20250115 02:53:30.741694 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cf8b9076fe465a99b51a0193993e76" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250" is_pre_election: true
I20250115 02:53:30.744383 11184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c59a2688344aa394e250fecdcafa33" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250" is_pre_election: true
W20250115 02:53:30.745596 11307 leader_election.cc:343] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:30.747651 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cf8b9076fe465a99b51a0193993e76" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250"
I20250115 02:53:30.744964 11389 raft_consensus.cc:357] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:30.749616 11389 raft_consensus.cc:383] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:30.749967 11389 raft_consensus.cc:738] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17d3d0a6e924761b7beb3110bf2481d, State: Initialized, Role: FOLLOWER
I20250115 02:53:30.750694 11389 consensus_queue.cc:260] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:30.755818 11389 ts_tablet_manager.cc:1428] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d: Time spent starting tablet: real 0.018s user 0.003s sys 0.003s
I20250115 02:53:30.731267 11388 tablet_bootstrap.cc:492] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250: No bootstrap required, opened a new log
I20250115 02:53:30.760219 11388 ts_tablet_manager.cc:1397] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250: Time spent bootstrapping tablet: real 0.123s user 0.014s sys 0.000s
W20250115 02:53:30.764622 11307 leader_election.cc:343] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): Illegal state: must be running to vote when last-logged opid is not known
W20250115 02:53:30.766837 11307 leader_election.cc:343] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:30.768081 11402 raft_consensus.cc:2798] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:30.768518 11402 raft_consensus.cc:491] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:30.768436 11306 leader_election.cc:304] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a17d3d0a6e924761b7beb3110bf2481d, cb6e4bf49c05498983e41c6e1da2c9da; no voters: 0c176e0eb85046bb98a030d22b3ae250
I20250115 02:53:30.769129 11402 raft_consensus.cc:3054] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:30.763998 11388 raft_consensus.cc:357] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:30.770051 11388 raft_consensus.cc:383] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:30.770362 11388 raft_consensus.cc:738] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c176e0eb85046bb98a030d22b3ae250, State: Initialized, Role: FOLLOWER
I20250115 02:53:30.772830 11417 raft_consensus.cc:2798] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:30.775067 11388 consensus_queue.cc:260] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:30.777456 11402 raft_consensus.cc:513] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:30.777915 11388 ts_tablet_manager.cc:1428] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250115 02:53:30.783028 11388 tablet_bootstrap.cc:492] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250: Bootstrap starting.
I20250115 02:53:30.803035 11417 raft_consensus.cc:695] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 LEADER]: Becoming Leader. State: Replica: cb6e4bf49c05498983e41c6e1da2c9da, State: Running, Role: LEADER
I20250115 02:53:30.801093 11388 tablet_bootstrap.cc:654] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:30.806298 11417 consensus_queue.cc:237] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:30.812712 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c59a2688344aa394e250fecdcafa33" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250"
I20250115 02:53:30.815796 11185 raft_consensus.cc:3054] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:30.822566 11262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c59a2688344aa394e250fecdcafa33" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17d3d0a6e924761b7beb3110bf2481d"
I20250115 02:53:30.823280 11262 raft_consensus.cc:3054] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:30.829725 11402 leader_election.cc:290] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 election: Requested vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955)
I20250115 02:53:30.872277 11262 raft_consensus.cc:2463] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6e4bf49c05498983e41c6e1da2c9da in term 1.
I20250115 02:53:30.875872 11185 raft_consensus.cc:2463] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6e4bf49c05498983e41c6e1da2c9da in term 1.
I20250115 02:53:30.875792 11388 tablet_bootstrap.cc:492] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250: No bootstrap required, opened a new log
I20250115 02:53:30.877765 11388 ts_tablet_manager.cc:1397] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250: Time spent bootstrapping tablet: real 0.095s user 0.005s sys 0.018s
I20250115 02:53:30.878062 11307 leader_election.cc:304] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c176e0eb85046bb98a030d22b3ae250, cb6e4bf49c05498983e41c6e1da2c9da; no voters:
I20250115 02:53:30.880349 11417 raft_consensus.cc:2798] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:30.881281 11417 raft_consensus.cc:695] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 LEADER]: Becoming Leader. State: Replica: cb6e4bf49c05498983e41c6e1da2c9da, State: Running, Role: LEADER
I20250115 02:53:30.880976 11388 raft_consensus.cc:357] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:30.887951 11388 raft_consensus.cc:383] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:30.888289 11388 raft_consensus.cc:738] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c176e0eb85046bb98a030d22b3ae250, State: Initialized, Role: FOLLOWER
I20250115 02:53:30.882391 11417 consensus_queue.cc:237] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:30.883967 10916 catalog_manager.cc:5526] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da reported cstate change: term changed from 0 to 1, leader changed from <none> to cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67). New cstate: current_term: 1 leader_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:30.924844 11388 consensus_queue.cc:260] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:30.925820 11403 raft_consensus.cc:491] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:30.926288 11403 raft_consensus.cc:513] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:30.930563 11403 leader_election.cc:290] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107), a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955)
I20250115 02:53:30.931941 11262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cf8b9076fe465a99b51a0193993e76" candidate_uuid: "0c176e0eb85046bb98a030d22b3ae250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17d3d0a6e924761b7beb3110bf2481d" is_pre_election: true
I20250115 02:53:30.932426 11341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cf8b9076fe465a99b51a0193993e76" candidate_uuid: "0c176e0eb85046bb98a030d22b3ae250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" is_pre_election: true
I20250115 02:53:30.932757 11262 raft_consensus.cc:2388] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c176e0eb85046bb98a030d22b3ae250 in current term 1: Already voted for candidate cb6e4bf49c05498983e41c6e1da2c9da in this term.
I20250115 02:53:30.943864 11388 ts_tablet_manager.cc:1428] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250: Time spent starting tablet: real 0.066s user 0.007s sys 0.010s
I20250115 02:53:30.965054 11141 leader_election.cc:304] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c176e0eb85046bb98a030d22b3ae250; no voters: a17d3d0a6e924761b7beb3110bf2481d, cb6e4bf49c05498983e41c6e1da2c9da
I20250115 02:53:30.967371 11403 raft_consensus.cc:3054] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:30.975050 11403 raft_consensus.cc:2743] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:53:30.975540 11404 raft_consensus.cc:491] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:30.976207 11404 raft_consensus.cc:513] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:30.980599 11404 leader_election.cc:290] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
I20250115 02:53:31.028949 11262 raft_consensus.cc:3054] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:31.049505 11403 consensus_queue.cc:1035] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 02:53:31.088626 10916 catalog_manager.cc:5526] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da reported cstate change: term changed from 0 to 1, leader changed from <none> to cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67). New cstate: current_term: 1 leader_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } health_report { overall_health: HEALTHY } } }
I20250115 02:53:31.125113 11407 consensus_queue.cc:1035] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 02:53:31.154072 11340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f3c76310ef4f0c84c12e6fd44fbe3b" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" is_pre_election: true
I20250115 02:53:31.225769 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f3c76310ef4f0c84c12e6fd44fbe3b" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250" is_pre_election: true
I20250115 02:53:31.228345 11229 leader_election.cc:304] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a17d3d0a6e924761b7beb3110bf2481d; no voters: 0c176e0eb85046bb98a030d22b3ae250, cb6e4bf49c05498983e41c6e1da2c9da
I20250115 02:53:31.229965 11404 raft_consensus.cc:2743] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:53:31.265899 10919 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47984:
name: "other-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "value"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\007\001d\000\000\000""\006\001\000\000\000\000\007\001d\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:53:31.269029 10919 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table other-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:53:31.323385 11173 tablet_service.cc:1467] Processing CreateTablet for tablet b4e67a5abbcc47218f661beb7684f551 (DEFAULT_TABLE table=other-table [id=95ff6ee622aa459dbf54d24997669042]), partition=RANGE (key) PARTITION 0 <= VALUES < 100
I20250115 02:53:31.323817 11252 tablet_service.cc:1467] Processing CreateTablet for tablet b4e67a5abbcc47218f661beb7684f551 (DEFAULT_TABLE table=other-table [id=95ff6ee622aa459dbf54d24997669042]), partition=RANGE (key) PARTITION 0 <= VALUES < 100
I20250115 02:53:31.324910 11173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e67a5abbcc47218f661beb7684f551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:31.325140 11252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e67a5abbcc47218f661beb7684f551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:31.325081 11331 tablet_service.cc:1467] Processing CreateTablet for tablet b4e67a5abbcc47218f661beb7684f551 (DEFAULT_TABLE table=other-table [id=95ff6ee622aa459dbf54d24997669042]), partition=RANGE (key) PARTITION 0 <= VALUES < 100
I20250115 02:53:31.326628 11331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e67a5abbcc47218f661beb7684f551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:31.351276 11417 consensus_queue.cc:1035] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:31.365720 11417 consensus_queue.cc:1035] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:31.432718 11388 tablet_bootstrap.cc:492] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250: Bootstrap starting.
I20250115 02:53:31.439716 11388 tablet_bootstrap.cc:654] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:31.454154 11441 tablet_bootstrap.cc:492] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d: Bootstrap starting.
I20250115 02:53:31.457028 11443 tablet_bootstrap.cc:492] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da: Bootstrap starting.
I20250115 02:53:31.469913 11443 tablet_bootstrap.cc:654] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:31.480000 11388 tablet_bootstrap.cc:492] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250: No bootstrap required, opened a new log
I20250115 02:53:31.486053 11447 consensus_queue.cc:1035] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 02:53:31.492580 11388 ts_tablet_manager.cc:1397] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.006s
I20250115 02:53:31.488890 11449 consensus_queue.cc:1035] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250115 02:53:31.511319 11441 tablet_bootstrap.cc:654] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:31.528909 11388 raft_consensus.cc:357] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:31.529743 11388 raft_consensus.cc:383] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:31.530046 11388 raft_consensus.cc:738] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c176e0eb85046bb98a030d22b3ae250, State: Initialized, Role: FOLLOWER
I20250115 02:53:31.530982 11388 consensus_queue.cc:260] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:31.534258 11388 ts_tablet_manager.cc:1428] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250: Time spent starting tablet: real 0.041s user 0.004s sys 0.003s
I20250115 02:53:31.536185 11443 tablet_bootstrap.cc:492] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da: No bootstrap required, opened a new log
I20250115 02:53:31.537271 11443 ts_tablet_manager.cc:1397] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da: Time spent bootstrapping tablet: real 0.080s user 0.021s sys 0.007s
I20250115 02:53:31.547617 11443 raft_consensus.cc:357] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:31.548602 11443 raft_consensus.cc:383] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:31.549172 11443 raft_consensus.cc:738] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e4bf49c05498983e41c6e1da2c9da, State: Initialized, Role: FOLLOWER
I20250115 02:53:31.550410 11443 consensus_queue.cc:260] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:31.556964 11443 ts_tablet_manager.cc:1428] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da: Time spent starting tablet: real 0.019s user 0.006s sys 0.002s
I20250115 02:53:31.570411 11441 tablet_bootstrap.cc:492] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d: No bootstrap required, opened a new log
I20250115 02:53:31.572312 11441 ts_tablet_manager.cc:1397] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d: Time spent bootstrapping tablet: real 0.118s user 0.010s sys 0.003s
I20250115 02:53:31.583643 11441 raft_consensus.cc:357] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:31.584446 11441 raft_consensus.cc:383] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:31.584739 11441 raft_consensus.cc:738] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17d3d0a6e924761b7beb3110bf2481d, State: Initialized, Role: FOLLOWER
I20250115 02:53:31.585536 11441 consensus_queue.cc:260] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:31.596839 11441 ts_tablet_manager.cc:1428] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20250115 02:53:31.917878 11417 raft_consensus.cc:491] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:31.918434 11417 raft_consensus.cc:513] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:31.920451 11417 leader_election.cc:290] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955)
I20250115 02:53:31.921674 11262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e67a5abbcc47218f661beb7684f551" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17d3d0a6e924761b7beb3110bf2481d" is_pre_election: true
I20250115 02:53:31.921903 11184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e67a5abbcc47218f661beb7684f551" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250" is_pre_election: true
I20250115 02:53:31.922411 11262 raft_consensus.cc:2463] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6e4bf49c05498983e41c6e1da2c9da in term 0.
I20250115 02:53:31.922596 11184 raft_consensus.cc:2463] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6e4bf49c05498983e41c6e1da2c9da in term 0.
I20250115 02:53:31.923818 11306 leader_election.cc:304] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a17d3d0a6e924761b7beb3110bf2481d, cb6e4bf49c05498983e41c6e1da2c9da; no voters:
I20250115 02:53:31.925137 11417 raft_consensus.cc:2798] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:31.925479 11417 raft_consensus.cc:491] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:31.925782 11417 raft_consensus.cc:3054] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:31.941619 11417 raft_consensus.cc:513] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:31.945003 11184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e67a5abbcc47218f661beb7684f551" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250"
I20250115 02:53:31.945631 11184 raft_consensus.cc:3054] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:31.947041 11262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e67a5abbcc47218f661beb7684f551" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17d3d0a6e924761b7beb3110bf2481d"
I20250115 02:53:31.947685 11262 raft_consensus.cc:3054] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:31.958420 11262 raft_consensus.cc:2463] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6e4bf49c05498983e41c6e1da2c9da in term 1.
I20250115 02:53:31.959360 11417 leader_election.cc:290] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 election: Requested vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955)
I20250115 02:53:31.959556 11306 leader_election.cc:304] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a17d3d0a6e924761b7beb3110bf2481d, cb6e4bf49c05498983e41c6e1da2c9da; no voters:
I20250115 02:53:31.960112 11184 raft_consensus.cc:2463] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6e4bf49c05498983e41c6e1da2c9da in term 1.
I20250115 02:53:31.961750 11417 raft_consensus.cc:2798] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:31.962563 11417 raft_consensus.cc:695] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 LEADER]: Becoming Leader. State: Replica: cb6e4bf49c05498983e41c6e1da2c9da, State: Running, Role: LEADER
I20250115 02:53:31.963423 11417 consensus_queue.cc:237] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:31.989943 10919 catalog_manager.cc:5526] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da reported cstate change: term changed from 0 to 1, leader changed from <none> to cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67). New cstate: current_term: 1 leader_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } health_report { overall_health: HEALTHY } } }
I20250115 02:53:32.151882 10919 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47984:
table { table_name: "other-table" } alter_schema_steps { type: DROP_RANGE_PARTITION drop_range_partition { range_bounds { rows: "\006\001\000\000\000\000\007\001d\000\000\000""\006\001\000\000\000\000\007\001d\000\000\000" indirect_data: """" } } } schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "value" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } modify_external_catalogs: true
I20250115 02:53:32.200322 10919 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3: Sending DeleteTablet for 3 replicas of tablet b4e67a5abbcc47218f661beb7684f551
I20250115 02:53:32.203158 11252 tablet_service.cc:1514] Processing DeleteTablet for tablet b4e67a5abbcc47218f661beb7684f551 with delete_type TABLET_DATA_DELETED (Partition dropped at 2025-01-15 02:53:32 UTC) from {username='slave'} at 127.0.0.1:58382
I20250115 02:53:32.207384 11331 tablet_service.cc:1514] Processing DeleteTablet for tablet b4e67a5abbcc47218f661beb7684f551 with delete_type TABLET_DATA_DELETED (Partition dropped at 2025-01-15 02:53:32 UTC) from {username='slave'} at 127.0.0.1:43564
I20250115 02:53:32.209939 11464 tablet_replica.cc:331] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da: stopping tablet replica
I20250115 02:53:32.211007 11464 raft_consensus.cc:2238] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:32.211020 11173 tablet_service.cc:1514] Processing DeleteTablet for tablet b4e67a5abbcc47218f661beb7684f551 with delete_type TABLET_DATA_DELETED (Partition dropped at 2025-01-15 02:53:32 UTC) from {username='slave'} at 127.0.0.1:58420
I20250115 02:53:32.221771 11464 pending_rounds.cc:62] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da: Trying to abort 1 pending ops.
I20250115 02:53:32.222978 11464 pending_rounds.cc:69] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114381770638999552 op_type: NO_OP noop_request { }
I20250115 02:53:32.249181 11463 tablet_replica.cc:331] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d: stopping tablet replica
I20250115 02:53:32.255939 11464 raft_consensus.cc:2883] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 02:53:32.271692 11464 raft_consensus.cc:2267] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:32.267575 11466 tablet_replica.cc:331] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250: stopping tablet replica
I20250115 02:53:32.284061 11464 ts_tablet_manager.cc:1905] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:32.283911 11463 raft_consensus.cc:2238] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:32.285123 11463 raft_consensus.cc:2267] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:32.284543 11466 raft_consensus.cc:2238] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:32.286809 11466 raft_consensus.cc:2267] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:32.289932 11463 ts_tablet_manager.cc:1905] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:32.336099 11466 ts_tablet_manager.cc:1905] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:32.347344 11463 ts_tablet_manager.cc:1918] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 02:53:32.392609 11463 log.cc:1198] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-1-root/wals/b4e67a5abbcc47218f661beb7684f551
I20250115 02:53:32.393452 11463 ts_tablet_manager.cc:1939] T b4e67a5abbcc47218f661beb7684f551 P a17d3d0a6e924761b7beb3110bf2481d: Deleting consensus metadata
I20250115 02:53:32.397413 11464 ts_tablet_manager.cc:1918] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 02:53:32.397792 11464 log.cc:1198] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-2-root/wals/b4e67a5abbcc47218f661beb7684f551
I20250115 02:53:32.410741 10905 catalog_manager.cc:4872] TS a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955): tablet b4e67a5abbcc47218f661beb7684f551 (table other-table [id=95ff6ee622aa459dbf54d24997669042]) successfully deleted
I20250115 02:53:32.413049 11464 ts_tablet_manager.cc:1939] T b4e67a5abbcc47218f661beb7684f551 P cb6e4bf49c05498983e41c6e1da2c9da: Deleting consensus metadata
I20250115 02:53:32.415867 10906 catalog_manager.cc:4872] TS cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107): tablet b4e67a5abbcc47218f661beb7684f551 (table other-table [id=95ff6ee622aa459dbf54d24997669042]) successfully deleted
I20250115 02:53:32.474364 11466 ts_tablet_manager.cc:1918] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 02:53:32.474884 11466 log.cc:1198] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestFilterOnNotabletTable.1736909525896969-2193-0/minicluster-data/ts-0-root/wals/b4e67a5abbcc47218f661beb7684f551
I20250115 02:53:32.475776 11466 ts_tablet_manager.cc:1939] T b4e67a5abbcc47218f661beb7684f551 P 0c176e0eb85046bb98a030d22b3ae250: Deleting consensus metadata
I20250115 02:53:32.485101 10906 catalog_manager.cc:4872] TS 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): tablet b4e67a5abbcc47218f661beb7684f551 (table other-table [id=95ff6ee622aa459dbf54d24997669042]) successfully deleted
W20250115 02:53:34.479169 11307 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.864s user 0.027s sys 0.040s
W20250115 02:53:34.763196 11481 sasl_common.cc:323] Time spent creating new SASL kudu client: real 0.291s user 0.001s sys 0.000s
W20250115 02:53:34.650213 11307 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.035s user 0.027s sys 0.040s
I20250115 02:53:34.971761 11485 raft_consensus.cc:491] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb6e4bf49c05498983e41c6e1da2c9da)
I20250115 02:53:34.972535 11485 raft_consensus.cc:513] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:34.974853 11485 leader_election.cc:290] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
W20250115 02:53:34.977520 11305 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.000s sys 0.012s
W20250115 02:53:35.260782 11305 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.005s user 0.000s sys 0.012s
I20250115 02:53:35.261679 11486 raft_consensus.cc:491] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c176e0eb85046bb98a030d22b3ae250)
I20250115 02:53:35.262328 11486 raft_consensus.cc:513] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:35.266177 11262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f3c76310ef4f0c84c12e6fd44fbe3b" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a17d3d0a6e924761b7beb3110bf2481d" is_pre_election: true
I20250115 02:53:35.267021 11486 leader_election.cc:290] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955)
W20250115 02:53:35.113474 11304 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.857s user 0.004s sys 0.005s
W20250115 02:53:35.267796 11304 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.242s user 0.005s sys 0.006s
I20250115 02:53:35.272116 11341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cf8b9076fe465a99b51a0193993e76" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" is_pre_election: true
W20250115 02:53:35.137782 11476 thread.cc:635] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.002s sys 0.005s
W20250115 02:53:35.274012 11476 thread.cc:601] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.807s user 0.002s sys 0.006s
I20250115 02:53:35.249322 11489 raft_consensus.cc:491] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c176e0eb85046bb98a030d22b3ae250)
I20250115 02:53:35.031898 11487 raft_consensus.cc:491] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb6e4bf49c05498983e41c6e1da2c9da)
I20250115 02:53:35.317823 11487 raft_consensus.cc:513] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:35.325551 11487 leader_election.cc:290] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
I20250115 02:53:35.327459 11341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c59a2688344aa394e250fecdcafa33" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" is_pre_election: true
I20250115 02:53:35.316664 11489 raft_consensus.cc:513] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:35.362792 11262 raft_consensus.cc:2463] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6e4bf49c05498983e41c6e1da2c9da in term 1.
I20250115 02:53:35.364411 11048 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35086 (request call id 134) took 2329 ms. Trace:
I20250115 02:53:35.370568 11489 leader_election.cc:290] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
I20250115 02:53:35.371099 11048 rpcz_store.cc:276] 0115 02:53:33.034933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:33.038545 (+ 3612us) service_pool.cc:225] Handling call
0115 02:53:35.364203 (+2325658us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 02:53:35.371641 11341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f3c76310ef4f0c84c12e6fd44fbe3b" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" is_pre_election: true
I20250115 02:53:35.372699 11341 raft_consensus.cc:2463] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a17d3d0a6e924761b7beb3110bf2481d in term 1.
I20250115 02:53:35.373952 11229 leader_election.cc:304] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a17d3d0a6e924761b7beb3110bf2481d, cb6e4bf49c05498983e41c6e1da2c9da; no voters:
I20250115 02:53:35.375175 11489 raft_consensus.cc:2798] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:53:35.375550 11489 raft_consensus.cc:491] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0c176e0eb85046bb98a030d22b3ae250)
I20250115 02:53:35.375903 11489 raft_consensus.cc:3054] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:35.383436 11489 raft_consensus.cc:513] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:35.387158 11341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f3c76310ef4f0c84c12e6fd44fbe3b" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb6e4bf49c05498983e41c6e1da2c9da"
I20250115 02:53:35.387827 11341 raft_consensus.cc:3054] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:35.395265 11341 raft_consensus.cc:2463] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a17d3d0a6e924761b7beb3110bf2481d in term 2.
I20250115 02:53:35.396972 11229 leader_election.cc:304] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a17d3d0a6e924761b7beb3110bf2481d, cb6e4bf49c05498983e41c6e1da2c9da; no voters:
I20250115 02:53:35.395766 11489 leader_election.cc:290] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 2 election: Requested vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
I20250115 02:53:35.398172 11489 raft_consensus.cc:2798] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 2 FOLLOWER]: Leader election won for term 2
I20250115 02:53:35.398830 11489 raft_consensus.cc:695] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 2 LEADER]: Becoming Leader. State: Replica: a17d3d0a6e924761b7beb3110bf2481d, State: Running, Role: LEADER
I20250115 02:53:35.399717 11489 consensus_queue.cc:237] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:35.560418 11049 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35098 (request call id 115) took 2525 ms. Trace:
I20250115 02:53:35.560755 11049 rpcz_store.cc:276] 0115 02:53:33.034564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:35.559455 (+2524891us) service_pool.cc:225] Handling call
0115 02:53:35.560292 (+ 837us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 02:53:35.563601 11492 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6d1251df61b4206b288016c970f9cb3)
I20250115 02:53:35.564167 11492 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d1251df61b4206b288016c970f9cb3" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 36515 } } peers { permanent_uuid: "9969c8550e5c4f829bc81536e86cff57" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 38335 } } peers { permanent_uuid: "8237eef53aa04c5296dcfcec9a7c3382" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 38535 } }
I20250115 02:53:35.567194 11492 leader_election.cc:290] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6d1251df61b4206b288016c970f9cb3 (127.2.36.126:36515), 9969c8550e5c4f829bc81536e86cff57 (127.2.36.125:38335)
W20250115 02:53:35.569797 10904 thread.cc:635] server-negotiator [worker] (thread pool) Time spent creating pthread: real 2.638s user 0.017s sys 0.023s
W20250115 02:53:35.570287 10904 thread.cc:601] server-negotiator [worker] (thread pool) Time spent starting thread: real 2.638s user 0.017s sys 0.024s
I20250115 02:53:35.594818 10993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8237eef53aa04c5296dcfcec9a7c3382" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "9969c8550e5c4f829bc81536e86cff57" is_pre_election: true
I20250115 02:53:35.622798 11497 raft_consensus.cc:491] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb6e4bf49c05498983e41c6e1da2c9da)
I20250115 02:53:35.623543 11497 raft_consensus.cc:513] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:35.625934 11497 leader_election.cc:290] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107), a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955)
I20250115 02:53:35.626549 11497 raft_consensus.cc:491] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb6e4bf49c05498983e41c6e1da2c9da)
I20250115 02:53:35.627153 11497 raft_consensus.cc:513] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:35.629180 11497 leader_election.cc:290] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66:43955), cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
I20250115 02:53:35.631265 11184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cf8b9076fe465a99b51a0193993e76" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250" is_pre_election: true
I20250115 02:53:35.631947 11184 raft_consensus.cc:2463] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a17d3d0a6e924761b7beb3110bf2481d in term 1.
I20250115 02:53:35.631942 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c59a2688344aa394e250fecdcafa33" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250" is_pre_election: true
I20250115 02:53:35.633481 11185 raft_consensus.cc:2463] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a17d3d0a6e924761b7beb3110bf2481d in term 1.
I20250115 02:53:35.633657 11229 leader_election.cc:304] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c176e0eb85046bb98a030d22b3ae250, a17d3d0a6e924761b7beb3110bf2481d; no voters: cb6e4bf49c05498983e41c6e1da2c9da
I20250115 02:53:35.632798 11183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f3c76310ef4f0c84c12e6fd44fbe3b" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250" is_pre_election: true
I20250115 02:53:35.634999 11489 raft_consensus.cc:2798] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:53:35.635390 11489 raft_consensus.cc:491] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Starting leader election (detected failure of leader cb6e4bf49c05498983e41c6e1da2c9da)
I20250115 02:53:35.635329 11184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f3c76310ef4f0c84c12e6fd44fbe3b" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250"
I20250115 02:53:35.635923 11489 raft_consensus.cc:3054] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:35.648573 11229 leader_election.cc:304] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c176e0eb85046bb98a030d22b3ae250, a17d3d0a6e924761b7beb3110bf2481d; no voters: cb6e4bf49c05498983e41c6e1da2c9da
I20250115 02:53:35.649556 11487 raft_consensus.cc:2798] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 02:53:35.649350 11489 raft_consensus.cc:513] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:35.649925 10916 catalog_manager.cc:5526] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d reported cstate change: term changed from 1 to 2, leader changed from 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65) to a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66). New cstate: current_term: 2 leader_uuid: "a17d3d0a6e924761b7beb3110bf2481d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:35.651448 11489 leader_election.cc:290] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 2 election: Requested vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
I20250115 02:53:35.649936 11487 raft_consensus.cc:491] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Starting leader election (detected failure of leader cb6e4bf49c05498983e41c6e1da2c9da)
I20250115 02:53:35.652330 11487 raft_consensus.cc:3054] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:35.657732 10984 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58302 (request call id 139) took 2622 ms. Trace:
I20250115 02:53:35.658134 10984 rpcz_store.cc:276] 0115 02:53:33.034745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:35.656303 (+2621558us) service_pool.cc:225] Handling call
0115 02:53:35.657560 (+ 1257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 02:53:35.660558 11184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cf8b9076fe465a99b51a0193993e76" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250"
I20250115 02:53:35.661176 11184 raft_consensus.cc:3054] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:35.661178 11487 raft_consensus.cc:513] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:35.663552 11487 leader_election.cc:290] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 2 election: Requested vote from peers 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759), cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67:33107)
I20250115 02:53:35.665310 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c59a2688344aa394e250fecdcafa33" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250"
I20250115 02:53:35.665781 11185 raft_consensus.cc:3054] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:35.668412 11184 raft_consensus.cc:2463] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a17d3d0a6e924761b7beb3110bf2481d in term 2.
I20250115 02:53:35.669493 11229 leader_election.cc:304] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c176e0eb85046bb98a030d22b3ae250, a17d3d0a6e924761b7beb3110bf2481d; no voters:
I20250115 02:53:35.670451 11487 raft_consensus.cc:2798] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 2 FOLLOWER]: Leader election won for term 2
I20250115 02:53:35.671283 11487 raft_consensus.cc:695] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 2 LEADER]: Becoming Leader. State: Replica: a17d3d0a6e924761b7beb3110bf2481d, State: Running, Role: LEADER
I20250115 02:53:35.672175 11487 consensus_queue.cc:237] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:35.673342 11183 raft_consensus.cc:1235] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Rejecting Update request from peer cb6e4bf49c05498983e41c6e1da2c9da for earlier term 1. Current term is 2. Ops: []
I20250115 02:53:35.674768 11182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f3c76310ef4f0c84c12e6fd44fbe3b" candidate_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c176e0eb85046bb98a030d22b3ae250" is_pre_election: true
I20250115 02:53:35.675537 11185 raft_consensus.cc:2463] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a17d3d0a6e924761b7beb3110bf2481d in term 2.
I20250115 02:53:35.676411 11263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cf8b9076fe465a99b51a0193993e76" candidate_uuid: "0c176e0eb85046bb98a030d22b3ae250" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a17d3d0a6e924761b7beb3110bf2481d" is_pre_election: true
I20250115 02:53:35.677819 11261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c59a2688344aa394e250fecdcafa33" candidate_uuid: "0c176e0eb85046bb98a030d22b3ae250" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a17d3d0a6e924761b7beb3110bf2481d" is_pre_election: true
I20250115 02:53:35.680714 11486 consensus_queue.cc:1046] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250115 02:53:35.681849 11484 raft_consensus.cc:3049] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 LEADER]: Stepping down as leader of term 1
I20250115 02:53:35.682193 11484 raft_consensus.cc:738] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb6e4bf49c05498983e41c6e1da2c9da, State: Running, Role: LEADER
I20250115 02:53:35.683367 11184 raft_consensus.cc:1235] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Rejecting Update request from peer cb6e4bf49c05498983e41c6e1da2c9da for earlier term 1. Current term is 2. Ops: []
I20250115 02:53:35.682960 11484 consensus_queue.cc:260] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } }
I20250115 02:53:35.687456 11484 raft_consensus.cc:3054] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:35.689687 11229 leader_election.cc:304] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c176e0eb85046bb98a030d22b3ae250, a17d3d0a6e924761b7beb3110bf2481d; no voters:
I20250115 02:53:35.690696 11489 raft_consensus.cc:2798] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 2 FOLLOWER]: Leader election won for term 2
I20250115 02:53:35.691226 11489 raft_consensus.cc:695] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 2 LEADER]: Becoming Leader. State: Replica: a17d3d0a6e924761b7beb3110bf2481d, State: Running, Role: LEADER
I20250115 02:53:35.692059 11489 consensus_queue.cc:237] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:35.698159 11484 consensus_queue.cc:1046] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250115 02:53:35.699455 11486 raft_consensus.cc:3049] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 LEADER]: Stepping down as leader of term 1
I20250115 02:53:35.699829 11486 raft_consensus.cc:738] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb6e4bf49c05498983e41c6e1da2c9da, State: Running, Role: LEADER
I20250115 02:53:35.700613 11486 consensus_queue.cc:260] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:35.703059 11338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cf8b9076fe465a99b51a0193993e76" candidate_uuid: "0c176e0eb85046bb98a030d22b3ae250" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" is_pre_election: true
I20250115 02:53:35.703846 11338 raft_consensus.cc:2463] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c176e0eb85046bb98a030d22b3ae250 in term 2.
I20250115 02:53:35.704559 11486 raft_consensus.cc:3054] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:35.708024 11339 raft_consensus.cc:1235] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Rejecting Update request from peer 0c176e0eb85046bb98a030d22b3ae250 for earlier term 1. Current term is 2. Ops: []
I20250115 02:53:35.712498 11340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c59a2688344aa394e250fecdcafa33" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb6e4bf49c05498983e41c6e1da2c9da"
I20250115 02:53:35.715199 11142 leader_election.cc:304] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c176e0eb85046bb98a030d22b3ae250, cb6e4bf49c05498983e41c6e1da2c9da; no voters: a17d3d0a6e924761b7beb3110bf2481d
I20250115 02:53:35.716187 11498 raft_consensus.cc:2758] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250115 02:53:35.716936 11498 consensus_queue.cc:1046] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250115 02:53:35.717903 11498 raft_consensus.cc:3049] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 1 LEADER]: Stepping down as leader of term 1
I20250115 02:53:35.718204 11498 raft_consensus.cc:738] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c176e0eb85046bb98a030d22b3ae250, State: Running, Role: LEADER
I20250115 02:53:35.718971 11498 consensus_queue.cc:260] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } }
I20250115 02:53:35.720156 11498 raft_consensus.cc:3054] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 1 FOLLOWER]: Advancing to term 2
I20250115 02:53:35.867187 11260 raft_consensus.cc:1235] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 2 LEADER]: Rejecting Update request from peer 0c176e0eb85046bb98a030d22b3ae250 for earlier term 1. Current term is 2. Ops: []
I20250115 02:53:35.868675 11340 raft_consensus.cc:2463] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a17d3d0a6e924761b7beb3110bf2481d in term 2.
I20250115 02:53:35.898162 11306 leader_election.cc:304] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a17d3d0a6e924761b7beb3110bf2481d, cb6e4bf49c05498983e41c6e1da2c9da; no voters: 0c176e0eb85046bb98a030d22b3ae250
I20250115 02:53:35.898975 11486 raft_consensus.cc:2758] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250115 02:53:35.920403 11337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c59a2688344aa394e250fecdcafa33" candidate_uuid: "0c176e0eb85046bb98a030d22b3ae250" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" is_pre_election: true
I20250115 02:53:35.923920 11142 leader_election.cc:304] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c176e0eb85046bb98a030d22b3ae250; no voters: a17d3d0a6e924761b7beb3110bf2481d, cb6e4bf49c05498983e41c6e1da2c9da
I20250115 02:53:35.924748 11498 raft_consensus.cc:2743] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 02:53:35.933738 11341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cf8b9076fe465a99b51a0193993e76" candidate_uuid: "a17d3d0a6e924761b7beb3110bf2481d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb6e4bf49c05498983e41c6e1da2c9da"
I20250115 02:53:35.940871 11341 raft_consensus.cc:2463] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a17d3d0a6e924761b7beb3110bf2481d in term 2.
I20250115 02:53:35.946096 10929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8237eef53aa04c5296dcfcec9a7c3382" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "e6d1251df61b4206b288016c970f9cb3" is_pre_election: true
I20250115 02:53:35.947873 11034 leader_election.cc:304] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8237eef53aa04c5296dcfcec9a7c3382; no voters: 9969c8550e5c4f829bc81536e86cff57, e6d1251df61b4206b288016c970f9cb3
I20250115 02:53:35.980387 10916 catalog_manager.cc:5526] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d reported cstate change: term changed from 1 to 2, leader changed from cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67) to a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66). New cstate: current_term: 2 leader_uuid: "a17d3d0a6e924761b7beb3110bf2481d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } health_report { overall_health: HEALTHY } } }
I20250115 02:53:35.982303 10916 catalog_manager.cc:5526] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d reported cstate change: term changed from 1 to 2, leader changed from cb6e4bf49c05498983e41c6e1da2c9da (127.2.36.67) to a17d3d0a6e924761b7beb3110bf2481d (127.2.36.66). New cstate: current_term: 2 leader_uuid: "a17d3d0a6e924761b7beb3110bf2481d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17d3d0a6e924761b7beb3110bf2481d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:35.999605 11341 raft_consensus.cc:1270] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Refusing update from remote peer a17d3d0a6e924761b7beb3110bf2481d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250115 02:53:36.001513 11487 consensus_queue.cc:1035] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250115 02:53:36.009296 11492 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 02:53:36.073642 11184 raft_consensus.cc:1270] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Refusing update from remote peer a17d3d0a6e924761b7beb3110bf2481d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250115 02:53:36.075215 11489 consensus_queue.cc:1035] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250115 02:53:36.129833 11341 raft_consensus.cc:1270] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Refusing update from remote peer a17d3d0a6e924761b7beb3110bf2481d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250115 02:53:36.131510 11489 consensus_queue.cc:1035] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250115 02:53:36.152990 11337 raft_consensus.cc:1270] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Refusing update from remote peer a17d3d0a6e924761b7beb3110bf2481d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250115 02:53:36.154793 11489 consensus_queue.cc:1035] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6e4bf49c05498983e41c6e1da2c9da" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250115 02:53:36.235858 11184 raft_consensus.cc:1270] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Refusing update from remote peer a17d3d0a6e924761b7beb3110bf2481d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250115 02:53:36.270469 11173 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:53:36.268030 11487 consensus_queue.cc:1035] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20250115 02:53:36.287628 11184 raft_consensus.cc:1270] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Refusing update from remote peer a17d3d0a6e924761b7beb3110bf2481d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250115 02:53:36.325018 11252 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250115 02:53:36.339457 11534 consensus_queue.cc:1035] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c176e0eb85046bb98a030d22b3ae250" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250115 02:53:36.368536 11331 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250115 02:53:36.639796 11206 debug-util.cc:398] Leaking SignalData structure 0x7b080007f600 after lost signal to thread 2196
I20250115 02:53:37.798895 11282 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:58854 (request call id 5) took 1070 ms. Trace:
I20250115 02:53:37.799388 11282 rpcz_store.cc:276] 0115 02:53:36.728829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 02:53:36.729004 (+ 175us) service_pool.cc:225] Handling call
0115 02:53:37.798794 (+1069790us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 02:53:37.817760 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:53:37.937252 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:37.938724 2193 tablet_replica.cc:331] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250: stopping tablet replica
I20250115 02:53:37.943590 2193 raft_consensus.cc:2238] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:37.944052 2193 raft_consensus.cc:2267] T 17cf8b9076fe465a99b51a0193993e76 P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:37.946106 2193 tablet_replica.cc:331] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250: stopping tablet replica
I20250115 02:53:37.953938 2193 raft_consensus.cc:2238] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:37.954591 2193 raft_consensus.cc:2267] T f8c59a2688344aa394e250fecdcafa33 P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:37.957019 2193 tablet_replica.cc:331] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250: stopping tablet replica
I20250115 02:53:37.957995 2193 raft_consensus.cc:2238] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:37.958431 2193 raft_consensus.cc:2267] T f5f3c76310ef4f0c84c12e6fd44fbe3b P 0c176e0eb85046bb98a030d22b3ae250 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250115 02:53:38.192916 11015 debug-util.cc:398] Leaking SignalData structure 0x7b080023efa0 after lost signal to thread 2196
W20250115 02:53:38.193867 11015 debug-util.cc:398] Leaking SignalData structure 0x7b080005dbe0 after lost signal to thread 10953
W20250115 02:53:38.194818 11015 debug-util.cc:398] Leaking SignalData structure 0x7b08001f0fa0 after lost signal to thread 11018
W20250115 02:53:38.195858 11015 debug-util.cc:398] Leaking SignalData structure 0x7b0800175be0 after lost signal to thread 11083
W20250115 02:53:38.200455 11015 debug-util.cc:398] Leaking SignalData structure 0x7b0800029980 after lost signal to thread 11287
W20250115 02:53:38.201722 11015 debug-util.cc:398] Leaking SignalData structure 0x7b0800149060 after lost signal to thread 11365
W20250115 02:53:38.216610 11229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43759: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20250115 02:53:38.240957 11229 consensus_peers.cc:487] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d -> Peer 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): Couldn't send request to peer 0c176e0eb85046bb98a030d22b3ae250. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43759: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250115 02:53:38.273329 11229 consensus_peers.cc:487] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d -> Peer 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): Couldn't send request to peer 0c176e0eb85046bb98a030d22b3ae250. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43759: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250115 02:53:38.310492 11229 consensus_peers.cc:487] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d -> Peer 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): Couldn't send request to peer 0c176e0eb85046bb98a030d22b3ae250. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43759: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250115 02:53:38.980172 2193 thread.cc:535] Waited for 1000ms trying to join with diag-logger (tid 11206)
W20250115 02:53:39.980654 2193 thread.cc:535] Waited for 2000ms trying to join with diag-logger (tid 11206)
W20250115 02:53:40.410130 11229 consensus_peers.cc:487] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d -> Peer 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): Couldn't send request to peer 0c176e0eb85046bb98a030d22b3ae250. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43759: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250115 02:53:40.509774 11229 consensus_peers.cc:487] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d -> Peer 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): Couldn't send request to peer 0c176e0eb85046bb98a030d22b3ae250. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43759: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250115 02:53:40.897737 11229 consensus_peers.cc:487] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d -> Peer 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): Couldn't send request to peer 0c176e0eb85046bb98a030d22b3ae250. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43759: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250115 02:53:40.981459 2193 thread.cc:535] Waited for 3000ms trying to join with diag-logger (tid 11206)
W20250115 02:53:41.981997 2193 thread.cc:535] Waited for 4000ms trying to join with diag-logger (tid 11206)
W20250115 02:53:42.737150 11229 consensus_peers.cc:487] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d -> Peer 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): Couldn't send request to peer 0c176e0eb85046bb98a030d22b3ae250. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43759: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250115 02:53:42.902185 11229 consensus_peers.cc:487] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d -> Peer 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): Couldn't send request to peer 0c176e0eb85046bb98a030d22b3ae250. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43759: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250115 02:53:42.982424 2193 thread.cc:535] Waited for 5000ms trying to join with diag-logger (tid 11206)
W20250115 02:53:43.281739 11229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43759: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20250115 02:53:43.458232 11229 consensus_peers.cc:487] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d -> Peer 0c176e0eb85046bb98a030d22b3ae250 (127.2.36.65:43759): Couldn't send request to peer 0c176e0eb85046bb98a030d22b3ae250. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:43759: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250115 02:53:43.982937 2193 thread.cc:535] Waited for 6000ms trying to join with diag-logger (tid 11206)
I20250115 02:53:44.506975 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:53:44.553225 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:53:44.648545 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:44.649597 2193 tablet_replica.cc:331] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d: stopping tablet replica
I20250115 02:53:44.650652 2193 raft_consensus.cc:2238] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 2 LEADER]: Raft consensus shutting down.
I20250115 02:53:44.652292 2193 raft_consensus.cc:2267] T f5f3c76310ef4f0c84c12e6fd44fbe3b P a17d3d0a6e924761b7beb3110bf2481d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:44.662216 2193 tablet_replica.cc:331] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d: stopping tablet replica
I20250115 02:53:44.662957 2193 raft_consensus.cc:2238] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 2 LEADER]: Raft consensus shutting down.
I20250115 02:53:44.664289 2193 raft_consensus.cc:2267] T 17cf8b9076fe465a99b51a0193993e76 P a17d3d0a6e924761b7beb3110bf2481d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:44.667647 2193 tablet_replica.cc:331] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d: stopping tablet replica
I20250115 02:53:44.668313 2193 raft_consensus.cc:2238] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 2 LEADER]: Raft consensus shutting down.
I20250115 02:53:44.669519 2193 raft_consensus.cc:2267] T f8c59a2688344aa394e250fecdcafa33 P a17d3d0a6e924761b7beb3110bf2481d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:44.693323 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:53:44.726794 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:53:44.809144 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:44.810071 2193 tablet_replica.cc:331] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da: stopping tablet replica
I20250115 02:53:44.810945 2193 raft_consensus.cc:2238] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:44.811614 2193 raft_consensus.cc:2267] T f5f3c76310ef4f0c84c12e6fd44fbe3b P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:44.815065 2193 tablet_replica.cc:331] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da: stopping tablet replica
I20250115 02:53:44.815752 2193 raft_consensus.cc:2238] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:44.816339 2193 raft_consensus.cc:2267] T f8c59a2688344aa394e250fecdcafa33 P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:44.819657 2193 tablet_replica.cc:331] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da: stopping tablet replica
I20250115 02:53:44.820330 2193 raft_consensus.cc:2238] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:44.820992 2193 raft_consensus.cc:2267] T 17cf8b9076fe465a99b51a0193993e76 P cb6e4bf49c05498983e41c6e1da2c9da [term 2 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:44.847728 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:53:44.875370 2193 master.cc:537] Master@127.2.36.126:36515 shutting down...
I20250115 02:53:44.915232 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:44.916534 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:44.917009 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P e6d1251df61b4206b288016c970f9cb3: stopping tablet replica
I20250115 02:53:45.087764 2193 master.cc:559] Master@127.2.36.126:36515 shutdown complete.
I20250115 02:53:45.105731 2193 master.cc:537] Master@127.2.36.125:38335 shutting down...
I20250115 02:53:45.129508 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:45.130121 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:45.130434 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 9969c8550e5c4f829bc81536e86cff57: stopping tablet replica
I20250115 02:53:45.175354 2193 master.cc:559] Master@127.2.36.125:38335 shutdown complete.
I20250115 02:53:45.190471 2193 master.cc:537] Master@127.2.36.124:38535 shutting down...
I20250115 02:53:45.210570 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:45.211243 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:45.211566 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 8237eef53aa04c5296dcfcec9a7c3382: stopping tablet replica
I20250115 02:53:45.230803 2193 master.cc:559] Master@127.2.36.124:38535 shutdown complete.
[ OK ] RemoteKsckTest.TestFilterOnNotabletTable (17053 ms)
[ RUN ] RemoteKsckTest.TestNonMatchingTabletIdFilter
I20250115 02:53:45.271783 2193 test_util.cc:274] Using random seed: -919034414
I20250115 02:53:45.272758 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:46643,127.2.36.125:39169,127.2.36.124:36523
I20250115 02:53:45.274000 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:45.279803 11556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:45.279930 11557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:45.280212 11559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:45.281430 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:45.282244 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:45.282440 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:45.282580 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909625282564 us; error 0 us; skew 500 ppm
I20250115 02:53:45.283119 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:45.285440 2193 webserver.cc:458] Webserver started at http://127.2.36.126:36333/ using document root <none> and password file <none>
I20250115 02:53:45.285885 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:45.286059 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:45.286306 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:45.287475 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "800b9250ed1942dc98e9a9684a27dcff"
format_stamp: "Formatted at 2025-01-15 02:53:45 on dist-test-slave-1lv2"
I20250115 02:53:45.291586 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250115 02:53:45.294737 11564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:45.295434 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:45.295692 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "800b9250ed1942dc98e9a9684a27dcff"
format_stamp: "Formatted at 2025-01-15 02:53:45 on dist-test-slave-1lv2"
I20250115 02:53:45.295943 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:45.322623 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:45.323771 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:45.359601 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:46643
I20250115 02:53:45.359702 11615 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:46643 every 8 connection(s)
I20250115 02:53:45.363263 11616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:45.363408 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:45.368790 11618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:45.369850 11619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:45.371526 11621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:45.372112 11616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:45.372347 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:45.373344 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:45.373575 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:45.373733 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909625373717 us; error 0 us; skew 500 ppm
I20250115 02:53:45.374297 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:45.377111 2193 webserver.cc:458] Webserver started at http://127.2.36.125:39733/ using document root <none> and password file <none>
I20250115 02:53:45.377557 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:45.377717 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:45.377948 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:45.379106 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "153981cfa51f4b1ea656612d7e0e4a45"
format_stamp: "Formatted at 2025-01-15 02:53:45 on dist-test-slave-1lv2"
I20250115 02:53:45.383760 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250115 02:53:45.387817 11628 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:45.388610 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:45.388877 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "153981cfa51f4b1ea656612d7e0e4a45"
format_stamp: "Formatted at 2025-01-15 02:53:45 on dist-test-slave-1lv2"
I20250115 02:53:45.389178 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:45.393054 11616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:45.398092 11616 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:39169: Network error: Client connection negotiation failed: client connection to 127.2.36.125:39169: connect: Connection refused (error 111)
I20250115 02:53:45.412938 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:45.414168 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:45.442607 11616 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } attempt: 1
W20250115 02:53:45.447202 11616 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:39169: Network error: Client connection negotiation failed: client connection to 127.2.36.125:39169: connect: Connection refused (error 111)
I20250115 02:53:45.453167 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:39169
I20250115 02:53:45.453258 11680 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:39169 every 8 connection(s)
I20250115 02:53:45.456820 11681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:45.456935 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:45.462507 11683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:45.462745 11684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:45.465001 11686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:45.465409 11681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:45.465927 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:45.466996 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:45.467291 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:45.467555 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909625467531 us; error 0 us; skew 500 ppm
I20250115 02:53:45.468194 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:45.470991 2193 webserver.cc:458] Webserver started at http://127.2.36.124:43237/ using document root <none> and password file <none>
I20250115 02:53:45.471479 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:45.471659 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:45.471963 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:45.473450 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7"
format_stamp: "Formatted at 2025-01-15 02:53:45 on dist-test-slave-1lv2"
I20250115 02:53:45.476907 11681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:45.478837 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20250115 02:53:45.483129 11693 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:45.484050 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250115 02:53:45.484407 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7"
format_stamp: "Formatted at 2025-01-15 02:53:45 on dist-test-slave-1lv2"
I20250115 02:53:45.484757 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:45.489183 11681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:45.494499 11681 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:36523: Network error: Client connection negotiation failed: client connection to 127.2.36.124:36523: connect: Connection refused (error 111)
I20250115 02:53:45.500710 11616 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } attempt: 2
I20250115 02:53:45.510254 11616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:45.510622 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:45.512051 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250115 02:53:45.514358 11616 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:36523: Network error: Client connection negotiation failed: client connection to 127.2.36.124:36523: connect: Connection refused (error 111)
I20250115 02:53:45.516009 11681 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } attempt: 1
W20250115 02:53:45.519786 11681 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:36523: Network error: Client connection negotiation failed: client connection to 127.2.36.124:36523: connect: Connection refused (error 111)
I20250115 02:53:45.549403 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:36523
I20250115 02:53:45.549499 11745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:36523 every 8 connection(s)
I20250115 02:53:45.552333 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:53:45.553282 11746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:45.559525 11746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:45.562883 11616 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } attempt: 1
I20250115 02:53:45.568317 11681 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } attempt: 2
I20250115 02:53:45.572188 11746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:45.581634 11746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:45.584100 11616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff: Bootstrap starting.
I20250115 02:53:45.588006 11681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45: Bootstrap starting.
I20250115 02:53:45.590287 11616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:45.593853 11681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:45.596330 11616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff: No bootstrap required, opened a new log
I20250115 02:53:45.598459 11681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45: No bootstrap required, opened a new log
I20250115 02:53:45.599578 11616 raft_consensus.cc:357] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } }
I20250115 02:53:45.600364 11616 raft_consensus.cc:383] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:45.600641 11616 raft_consensus.cc:738] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800b9250ed1942dc98e9a9684a27dcff, State: Initialized, Role: FOLLOWER
I20250115 02:53:45.600996 11746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7: Bootstrap starting.
I20250115 02:53:45.600811 11681 raft_consensus.cc:357] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } }
I20250115 02:53:45.601552 11681 raft_consensus.cc:383] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:45.601853 11681 raft_consensus.cc:738] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153981cfa51f4b1ea656612d7e0e4a45, State: Initialized, Role: FOLLOWER
I20250115 02:53:45.601390 11616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } }
I20250115 02:53:45.602447 11681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } }
I20250115 02:53:45.603790 11753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } } }
I20250115 02:53:45.604748 11753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:45.605508 11616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:45.607838 11746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:45.607594 11754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } } }
I20250115 02:53:45.608529 11754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:45.609426 11681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:45.618278 11746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7: No bootstrap required, opened a new log
I20250115 02:53:45.621412 11746 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } }
I20250115 02:53:45.622323 11746 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:45.622673 11746 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e67cb733f1a43b59140c0fcb8e1f9d7, State: Initialized, Role: FOLLOWER
I20250115 02:53:45.623337 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:53:45.623539 11746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } }
I20250115 02:53:45.626188 11773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } } }
I20250115 02:53:45.627074 11773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:45.627962 11746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [sys.catalog]: configured and running, proceeding with master startup.
W20250115 02:53:45.628337 11771 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:45.628751 11771 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:45.631103 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250115 02:53:45.635701 11778 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:45.636073 11778 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:45.641016 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:45.642848 11789 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:45.643131 11789 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:53:45.647302 11790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:45.648128 11791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:45.650267 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:45.650493 11793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:45.651381 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:45.651582 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:45.651718 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909625651707 us; error 0 us; skew 500 ppm
I20250115 02:53:45.652179 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:45.654356 2193 webserver.cc:458] Webserver started at http://127.2.36.65:34375/ using document root <none> and password file <none>
I20250115 02:53:45.654822 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:45.654982 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:45.655191 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:45.656241 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "cf6d85313a5347e8bc7f4e91ead0c855"
format_stamp: "Formatted at 2025-01-15 02:53:45 on dist-test-slave-1lv2"
I20250115 02:53:45.660455 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:53:45.663405 11798 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:45.664109 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:45.664355 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "cf6d85313a5347e8bc7f4e91ead0c855"
format_stamp: "Formatted at 2025-01-15 02:53:45 on dist-test-slave-1lv2"
I20250115 02:53:45.664577 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250115 02:53:45.667037 11635 tablet.cc:2367] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250115 02:53:45.682446 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:45.683686 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:45.685024 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:45.687425 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:45.687618 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:45.687857 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:45.688004 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:45.701819 11754 raft_consensus.cc:491] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:45.702304 11754 raft_consensus.cc:513] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } }
I20250115 02:53:45.704082 11754 leader_election.cc:290] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 800b9250ed1942dc98e9a9684a27dcff (127.2.36.126:46643), 8e67cb733f1a43b59140c0fcb8e1f9d7 (127.2.36.124:36523)
I20250115 02:53:45.705111 11591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "153981cfa51f4b1ea656612d7e0e4a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "800b9250ed1942dc98e9a9684a27dcff" is_pre_election: true
I20250115 02:53:45.705626 11721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "153981cfa51f4b1ea656612d7e0e4a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" is_pre_election: true
I20250115 02:53:45.706063 11591 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153981cfa51f4b1ea656612d7e0e4a45 in term 0.
I20250115 02:53:45.706297 11721 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153981cfa51f4b1ea656612d7e0e4a45 in term 0.
I20250115 02:53:45.707587 11629 leader_election.cc:304] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153981cfa51f4b1ea656612d7e0e4a45, 800b9250ed1942dc98e9a9684a27dcff; no voters:
I20250115 02:53:45.708844 11754 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:45.709214 11754 raft_consensus.cc:491] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:45.709563 11754 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:45.717584 11754 raft_consensus.cc:513] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } }
I20250115 02:53:45.719772 11754 leader_election.cc:290] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [CANDIDATE]: Term 1 election: Requested vote from peers 800b9250ed1942dc98e9a9684a27dcff (127.2.36.126:46643), 8e67cb733f1a43b59140c0fcb8e1f9d7 (127.2.36.124:36523)
I20250115 02:53:45.720547 11591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "153981cfa51f4b1ea656612d7e0e4a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "800b9250ed1942dc98e9a9684a27dcff"
I20250115 02:53:45.721186 11591 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:45.721375 11721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "153981cfa51f4b1ea656612d7e0e4a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7"
I20250115 02:53:45.722003 11721 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:45.727083 11591 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153981cfa51f4b1ea656612d7e0e4a45 in term 1.
I20250115 02:53:45.728396 11629 leader_election.cc:304] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153981cfa51f4b1ea656612d7e0e4a45, 800b9250ed1942dc98e9a9684a27dcff; no voters:
I20250115 02:53:45.729341 11754 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:45.729566 11721 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153981cfa51f4b1ea656612d7e0e4a45 in term 1.
I20250115 02:53:45.731415 11754 raft_consensus.cc:695] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 1 LEADER]: Becoming Leader. State: Replica: 153981cfa51f4b1ea656612d7e0e4a45, State: Running, Role: LEADER
I20250115 02:53:45.732239 11754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } }
I20250115 02:53:45.737525 11853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 153981cfa51f4b1ea656612d7e0e4a45. Latest consensus state: current_term: 1 leader_uuid: "153981cfa51f4b1ea656612d7e0e4a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } } }
I20250115 02:53:45.738134 11853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:45.739012 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:33647
I20250115 02:53:45.739220 11862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:33647 every 8 connection(s)
I20250115 02:53:45.739848 11863 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:45.746021 11863 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:45.763129 11591 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [term 1 FOLLOWER]: Refusing update from remote peer 153981cfa51f4b1ea656612d7e0e4a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:45.763129 11721 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [term 1 FOLLOWER]: Refusing update from remote peer 153981cfa51f4b1ea656612d7e0e4a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:45.764930 11754 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:45.769284 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:45.766513 11754 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250115 02:53:45.774508 11865 heartbeater.cc:346] Connected to a master server at 127.2.36.124:36523
I20250115 02:53:45.774933 11865 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:45.775784 11865 heartbeater.cc:509] Master 127.2.36.124:36523 requested a full tablet report, sending...
I20250115 02:53:45.780676 11864 heartbeater.cc:346] Connected to a master server at 127.2.36.125:39169
I20250115 02:53:45.781077 11864 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:45.782020 11864 heartbeater.cc:509] Master 127.2.36.125:39169 requested a full tablet report, sending...
I20250115 02:53:45.784816 11646 ts_manager.cc:194] Registered new tserver with Master: cf6d85313a5347e8bc7f4e91ead0c855 (127.2.36.65:33647)
I20250115 02:53:45.803033 11866 heartbeater.cc:346] Connected to a master server at 127.2.36.126:46643
I20250115 02:53:45.803577 11866 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:45.804481 11866 heartbeater.cc:509] Master 127.2.36.126:46643 requested a full tablet report, sending...
I20250115 02:53:45.810688 11711 ts_manager.cc:194] Registered new tserver with Master: cf6d85313a5347e8bc7f4e91ead0c855 (127.2.36.65:33647)
I20250115 02:53:45.812614 11773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 153981cfa51f4b1ea656612d7e0e4a45. Latest consensus state: current_term: 1 leader_uuid: "153981cfa51f4b1ea656612d7e0e4a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } } }
I20250115 02:53:45.813475 11773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [sys.catalog]: This master's current role is: FOLLOWER
W20250115 02:53:45.813419 11874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:45.817828 11853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "153981cfa51f4b1ea656612d7e0e4a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } } }
I20250115 02:53:45.818640 11853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:45.820837 11863 catalog_manager.cc:1348] Generated new cluster ID: 0b9604fb31b246bfa84af80356b2feb0
I20250115 02:53:45.821209 11863 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:45.824594 11853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "153981cfa51f4b1ea656612d7e0e4a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } } }
I20250115 02:53:45.825407 11853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:45.833016 11581 ts_manager.cc:194] Registered new tserver with Master: cf6d85313a5347e8bc7f4e91ead0c855 (127.2.36.65:33647)
I20250115 02:53:45.834724 11753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 153981cfa51f4b1ea656612d7e0e4a45. Latest consensus state: current_term: 1 leader_uuid: "153981cfa51f4b1ea656612d7e0e4a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } } }
W20250115 02:53:45.835119 11877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:45.835611 11753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:45.839486 11773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "153981cfa51f4b1ea656612d7e0e4a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } } }
I20250115 02:53:45.840394 11773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:45.845152 11753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "153981cfa51f4b1ea656612d7e0e4a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b9250ed1942dc98e9a9684a27dcff" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 46643 } } peers { permanent_uuid: "153981cfa51f4b1ea656612d7e0e4a45" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 39169 } } peers { permanent_uuid: "8e67cb733f1a43b59140c0fcb8e1f9d7" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 36523 } } }
I20250115 02:53:45.845890 11753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [sys.catalog]: This master's current role is: FOLLOWER
W20250115 02:53:45.855530 11883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:45.855618 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:45.856935 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:45.857272 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:45.857547 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909625857527 us; error 0 us; skew 500 ppm
I20250115 02:53:45.858178 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:45.862342 11863 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:53:45.864025 11863 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:45.874181 2193 webserver.cc:458] Webserver started at http://127.2.36.66:38917/ using document root <none> and password file <none>
I20250115 02:53:45.874742 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:45.874961 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:45.875207 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:45.876210 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "575a4ac561734433a144a2958a5a98e2"
format_stamp: "Formatted at 2025-01-15 02:53:45 on dist-test-slave-1lv2"
I20250115 02:53:45.881510 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250115 02:53:45.885869 11888 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:45.890056 2193 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20250115 02:53:45.890514 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "575a4ac561734433a144a2958a5a98e2"
format_stamp: "Formatted at 2025-01-15 02:53:45 on dist-test-slave-1lv2"
I20250115 02:53:45.890980 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:45.895219 11863 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45: Generated new TSK 0
I20250115 02:53:45.896054 11863 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:45.909098 11646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43702
I20250115 02:53:45.918187 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:45.919492 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:45.920853 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:45.921339 11864 heartbeater.cc:501] Master 127.2.36.125:39169 was elected leader, sending a full tablet report...
I20250115 02:53:45.923368 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:45.923614 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:45.923867 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:45.924079 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:45.969739 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:34489
I20250115 02:53:45.969839 11951 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:34489 every 8 connection(s)
I20250115 02:53:45.986208 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:46.001010 11953 heartbeater.cc:346] Connected to a master server at 127.2.36.124:36523
I20250115 02:53:46.001502 11953 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:46.002506 11953 heartbeater.cc:509] Master 127.2.36.124:36523 requested a full tablet report, sending...
W20250115 02:53:46.001488 11959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:46.006086 11711 ts_manager.cc:194] Registered new tserver with Master: 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489)
W20250115 02:53:46.010916 11960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:46.018602 11952 heartbeater.cc:346] Connected to a master server at 127.2.36.125:39169
I20250115 02:53:46.019402 11952 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:46.020381 11952 heartbeater.cc:509] Master 127.2.36.125:39169 requested a full tablet report, sending...
I20250115 02:53:46.022557 11646 ts_manager.cc:194] Registered new tserver with Master: 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489)
I20250115 02:53:46.023176 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:46.024542 11646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43708
W20250115 02:53:46.026276 11963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:46.026523 11954 heartbeater.cc:346] Connected to a master server at 127.2.36.126:46643
I20250115 02:53:46.027066 11954 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:46.027426 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:46.027721 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:46.027724 11954 heartbeater.cc:509] Master 127.2.36.126:46643 requested a full tablet report, sending...
I20250115 02:53:46.028095 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909626028073 us; error 0 us; skew 500 ppm
I20250115 02:53:46.028962 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:46.030356 11581 ts_manager.cc:194] Registered new tserver with Master: 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489)
I20250115 02:53:46.032524 2193 webserver.cc:458] Webserver started at http://127.2.36.67:40437/ using document root <none> and password file <none>
I20250115 02:53:46.033025 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:46.033183 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:46.033391 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:46.034420 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95"
format_stamp: "Formatted at 2025-01-15 02:53:46 on dist-test-slave-1lv2"
I20250115 02:53:46.037389 11952 heartbeater.cc:501] Master 127.2.36.125:39169 was elected leader, sending a full tablet report...
I20250115 02:53:46.041244 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250115 02:53:46.045431 11967 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:46.046223 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:53:46.046525 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95"
format_stamp: "Formatted at 2025-01-15 02:53:46 on dist-test-slave-1lv2"
I20250115 02:53:46.046819 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestNonMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:46.064301 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:46.065961 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:46.068256 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:46.071089 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:46.071278 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:46.071513 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:46.071660 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:46.132334 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:37089
I20250115 02:53:46.132407 12029 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:37089 every 8 connection(s)
I20250115 02:53:46.161788 12031 heartbeater.cc:346] Connected to a master server at 127.2.36.124:36523
I20250115 02:53:46.162225 12031 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:46.163192 12031 heartbeater.cc:509] Master 127.2.36.124:36523 requested a full tablet report, sending...
I20250115 02:53:46.163787 12030 heartbeater.cc:346] Connected to a master server at 127.2.36.125:39169
I20250115 02:53:46.164250 12030 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:46.165124 12030 heartbeater.cc:509] Master 127.2.36.125:39169 requested a full tablet report, sending...
I20250115 02:53:46.167166 11711 ts_manager.cc:194] Registered new tserver with Master: f546b9ebad5f44aa8afa2e3d62e3ca95 (127.2.36.67:37089)
I20250115 02:53:46.170722 12032 heartbeater.cc:346] Connected to a master server at 127.2.36.126:46643
I20250115 02:53:46.171360 12032 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:46.172147 12032 heartbeater.cc:509] Master 127.2.36.126:46643 requested a full tablet report, sending...
I20250115 02:53:46.172914 11645 ts_manager.cc:194] Registered new tserver with Master: f546b9ebad5f44aa8afa2e3d62e3ca95 (127.2.36.67:37089)
I20250115 02:53:46.174459 11645 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43722
I20250115 02:53:46.174539 11580 ts_manager.cc:194] Registered new tserver with Master: f546b9ebad5f44aa8afa2e3d62e3ca95 (127.2.36.67:37089)
I20250115 02:53:46.174979 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.031794573s
I20250115 02:53:46.189425 12030 heartbeater.cc:501] Master 127.2.36.125:39169 was elected leader, sending a full tablet report...
I20250115 02:53:46.240006 11645 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43732:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:53:46.242779 11645 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:53:46.351375 11995 tablet_service.cc:1467] Processing CreateTablet for tablet 52fcbfa352bd4f0f9554a6e1d358e054 (DEFAULT_TABLE table=ksck-test-table [id=f1e28090d37d424c85e2e7bf0319e45d]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:46.351642 11994 tablet_service.cc:1467] Processing CreateTablet for tablet e66df73085f843aeacb448f6a6036e19 (DEFAULT_TABLE table=ksck-test-table [id=f1e28090d37d424c85e2e7bf0319e45d]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:46.352824 11995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52fcbfa352bd4f0f9554a6e1d358e054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:46.353525 11994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e66df73085f843aeacb448f6a6036e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:46.358604 11993 tablet_service.cc:1467] Processing CreateTablet for tablet c8e9503376e842b790858102bb4d8f1b (DEFAULT_TABLE table=ksck-test-table [id=f1e28090d37d424c85e2e7bf0319e45d]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:46.359959 11993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e9503376e842b790858102bb4d8f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:46.385061 12051 tablet_bootstrap.cc:492] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95: Bootstrap starting.
I20250115 02:53:46.390529 11825 tablet_service.cc:1467] Processing CreateTablet for tablet e66df73085f843aeacb448f6a6036e19 (DEFAULT_TABLE table=ksck-test-table [id=f1e28090d37d424c85e2e7bf0319e45d]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:46.392007 11825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e66df73085f843aeacb448f6a6036e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:46.393148 11916 tablet_service.cc:1467] Processing CreateTablet for tablet e66df73085f843aeacb448f6a6036e19 (DEFAULT_TABLE table=ksck-test-table [id=f1e28090d37d424c85e2e7bf0319e45d]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:46.394451 11916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e66df73085f843aeacb448f6a6036e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:46.395701 11915 tablet_service.cc:1467] Processing CreateTablet for tablet c8e9503376e842b790858102bb4d8f1b (DEFAULT_TABLE table=ksck-test-table [id=f1e28090d37d424c85e2e7bf0319e45d]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:46.398121 11824 tablet_service.cc:1467] Processing CreateTablet for tablet c8e9503376e842b790858102bb4d8f1b (DEFAULT_TABLE table=ksck-test-table [id=f1e28090d37d424c85e2e7bf0319e45d]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:46.399466 11824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e9503376e842b790858102bb4d8f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:46.402640 11826 tablet_service.cc:1467] Processing CreateTablet for tablet 52fcbfa352bd4f0f9554a6e1d358e054 (DEFAULT_TABLE table=ksck-test-table [id=f1e28090d37d424c85e2e7bf0319e45d]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:46.391701 11917 tablet_service.cc:1467] Processing CreateTablet for tablet 52fcbfa352bd4f0f9554a6e1d358e054 (DEFAULT_TABLE table=ksck-test-table [id=f1e28090d37d424c85e2e7bf0319e45d]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:46.407259 11826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52fcbfa352bd4f0f9554a6e1d358e054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:46.415985 12051 tablet_bootstrap.cc:654] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:46.416633 11915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e9503376e842b790858102bb4d8f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:46.419201 11917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52fcbfa352bd4f0f9554a6e1d358e054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:46.421368 12051 tablet_bootstrap.cc:492] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95: No bootstrap required, opened a new log
I20250115 02:53:46.421725 12051 ts_tablet_manager.cc:1397] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.004s
I20250115 02:53:46.425619 12051 raft_consensus.cc:357] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } }
I20250115 02:53:46.426366 12051 raft_consensus.cc:383] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:46.426652 12051 raft_consensus.cc:738] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f546b9ebad5f44aa8afa2e3d62e3ca95, State: Initialized, Role: FOLLOWER
I20250115 02:53:46.427421 12051 consensus_queue.cc:260] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } }
I20250115 02:53:46.430701 12051 ts_tablet_manager.cc:1428] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250115 02:53:46.432106 12051 tablet_bootstrap.cc:492] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95: Bootstrap starting.
I20250115 02:53:46.437999 12051 tablet_bootstrap.cc:654] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:46.449187 12051 tablet_bootstrap.cc:492] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95: No bootstrap required, opened a new log
I20250115 02:53:46.449697 12051 ts_tablet_manager.cc:1397] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20250115 02:53:46.452487 12051 raft_consensus.cc:357] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.453210 12051 raft_consensus.cc:383] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:46.453462 12051 raft_consensus.cc:738] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f546b9ebad5f44aa8afa2e3d62e3ca95, State: Initialized, Role: FOLLOWER
I20250115 02:53:46.454263 12051 consensus_queue.cc:260] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.456204 12051 ts_tablet_manager.cc:1428] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250115 02:53:46.459631 12051 tablet_bootstrap.cc:492] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95: Bootstrap starting.
I20250115 02:53:46.465623 12051 tablet_bootstrap.cc:654] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:46.469291 12053 tablet_bootstrap.cc:492] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855: Bootstrap starting.
I20250115 02:53:46.475579 12053 tablet_bootstrap.cc:654] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:46.480190 12051 tablet_bootstrap.cc:492] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95: No bootstrap required, opened a new log
I20250115 02:53:46.480605 12051 ts_tablet_manager.cc:1397] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20250115 02:53:46.482690 12051 raft_consensus.cc:357] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.483317 12051 raft_consensus.cc:383] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:46.483537 12051 raft_consensus.cc:738] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f546b9ebad5f44aa8afa2e3d62e3ca95, State: Initialized, Role: FOLLOWER
I20250115 02:53:46.484066 12051 consensus_queue.cc:260] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.486151 12051 ts_tablet_manager.cc:1428] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250115 02:53:46.499922 12053 tablet_bootstrap.cc:492] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855: No bootstrap required, opened a new log
I20250115 02:53:46.500486 12053 ts_tablet_manager.cc:1397] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.005s
I20250115 02:53:46.503753 12053 raft_consensus.cc:357] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.508378 12053 raft_consensus.cc:383] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:46.508693 12053 raft_consensus.cc:738] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6d85313a5347e8bc7f4e91ead0c855, State: Initialized, Role: FOLLOWER
I20250115 02:53:46.509476 12053 consensus_queue.cc:260] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.520304 12053 ts_tablet_manager.cc:1428] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855: Time spent starting tablet: real 0.019s user 0.011s sys 0.003s
I20250115 02:53:46.527865 12058 raft_consensus.cc:491] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:46.528445 12058 raft_consensus.cc:513] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.531104 12058 leader_election.cc:290] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489), f546b9ebad5f44aa8afa2e3d62e3ca95 (127.2.36.67:37089)
I20250115 02:53:46.548949 12053 tablet_bootstrap.cc:492] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855: Bootstrap starting.
W20250115 02:53:46.554651 11799 leader_election.cc:343] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489): Not found: Tablet not found: e66df73085f843aeacb448f6a6036e19
I20250115 02:53:46.561094 12053 tablet_bootstrap.cc:654] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:46.565749 12061 tablet_bootstrap.cc:492] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2: Bootstrap starting.
I20250115 02:53:46.569046 12053 tablet_bootstrap.cc:492] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855: No bootstrap required, opened a new log
I20250115 02:53:46.569648 12053 ts_tablet_manager.cc:1397] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.004s
I20250115 02:53:46.572470 12053 raft_consensus.cc:357] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.573247 12053 raft_consensus.cc:383] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:46.573077 12005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66df73085f843aeacb448f6a6036e19" candidate_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" is_pre_election: true
I20250115 02:53:46.573621 12053 raft_consensus.cc:738] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6d85313a5347e8bc7f4e91ead0c855, State: Initialized, Role: FOLLOWER
I20250115 02:53:46.573858 12005 raft_consensus.cc:2463] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6d85313a5347e8bc7f4e91ead0c855 in term 0.
I20250115 02:53:46.574343 12053 consensus_queue.cc:260] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.575330 11800 leader_election.cc:304] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6d85313a5347e8bc7f4e91ead0c855, f546b9ebad5f44aa8afa2e3d62e3ca95; no voters: 575a4ac561734433a144a2958a5a98e2
I20250115 02:53:46.576759 12053 ts_tablet_manager.cc:1428] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20250115 02:53:46.577155 12058 raft_consensus.cc:2798] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:46.579053 12058 raft_consensus.cc:491] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:46.579442 12058 raft_consensus.cc:3054] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:46.582861 12061 tablet_bootstrap.cc:654] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:46.577836 12053 tablet_bootstrap.cc:492] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855: Bootstrap starting.
I20250115 02:53:46.586577 12058 raft_consensus.cc:513] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.594619 11927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66df73085f843aeacb448f6a6036e19" candidate_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a4ac561734433a144a2958a5a98e2"
W20250115 02:53:46.596184 11799 leader_election.cc:343] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:46.596372 12063 raft_consensus.cc:491] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:46.597203 12058 leader_election.cc:290] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 election: Requested vote from peers 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489), f546b9ebad5f44aa8afa2e3d62e3ca95 (127.2.36.67:37089)
I20250115 02:53:46.598166 12005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66df73085f843aeacb448f6a6036e19" candidate_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95"
I20250115 02:53:46.600893 12005 raft_consensus.cc:3054] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:46.603399 12053 tablet_bootstrap.cc:654] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:46.613659 12005 raft_consensus.cc:2463] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6d85313a5347e8bc7f4e91ead0c855 in term 1.
I20250115 02:53:46.596976 12063 raft_consensus.cc:513] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.617666 11927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e9503376e842b790858102bb4d8f1b" candidate_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a4ac561734433a144a2958a5a98e2" is_pre_election: true
I20250115 02:53:46.618045 11800 leader_election.cc:304] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6d85313a5347e8bc7f4e91ead0c855, f546b9ebad5f44aa8afa2e3d62e3ca95; no voters: 575a4ac561734433a144a2958a5a98e2
I20250115 02:53:46.618956 12058 raft_consensus.cc:2798] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:46.619915 12053 tablet_bootstrap.cc:492] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855: No bootstrap required, opened a new log
I20250115 02:53:46.620638 12053 ts_tablet_manager.cc:1397] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.010s
W20250115 02:53:46.623800 11799 leader_election.cc:343] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:46.627300 12063 leader_election.cc:290] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489), f546b9ebad5f44aa8afa2e3d62e3ca95 (127.2.36.67:37089)
I20250115 02:53:46.630234 12058 raft_consensus.cc:695] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 LEADER]: Becoming Leader. State: Replica: cf6d85313a5347e8bc7f4e91ead0c855, State: Running, Role: LEADER
I20250115 02:53:46.631202 12058 consensus_queue.cc:237] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.633270 12053 raft_consensus.cc:357] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } }
I20250115 02:53:46.633850 12053 raft_consensus.cc:383] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:46.634032 12053 raft_consensus.cc:738] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6d85313a5347e8bc7f4e91ead0c855, State: Initialized, Role: FOLLOWER
I20250115 02:53:46.636916 12061 tablet_bootstrap.cc:492] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2: No bootstrap required, opened a new log
I20250115 02:53:46.637357 12061 ts_tablet_manager.cc:1397] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2: Time spent bootstrapping tablet: real 0.072s user 0.016s sys 0.009s
I20250115 02:53:46.639780 12005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e9503376e842b790858102bb4d8f1b" candidate_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" is_pre_election: true
I20250115 02:53:46.640197 12061 raft_consensus.cc:357] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.634735 11771 catalog_manager.cc:1260] Loaded cluster ID: 0b9604fb31b246bfa84af80356b2feb0
I20250115 02:53:46.641141 11771 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff: loading cluster ID for follower catalog manager: success
I20250115 02:53:46.642256 12061 raft_consensus.cc:383] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:46.643141 12005 raft_consensus.cc:2463] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6d85313a5347e8bc7f4e91ead0c855 in term 0.
I20250115 02:53:46.645032 11643 catalog_manager.cc:5526] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf6d85313a5347e8bc7f4e91ead0c855 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:46.648653 11789 catalog_manager.cc:1260] Loaded cluster ID: 0b9604fb31b246bfa84af80356b2feb0
I20250115 02:53:46.648952 11789 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7: loading cluster ID for follower catalog manager: success
I20250115 02:53:46.634667 12053 consensus_queue.cc:260] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } }
I20250115 02:53:46.655850 11789 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7: acquiring CA information for follower catalog manager: success
I20250115 02:53:46.657114 12053 ts_tablet_manager.cc:1428] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855: Time spent starting tablet: real 0.036s user 0.004s sys 0.002s
I20250115 02:53:46.657979 12061 raft_consensus.cc:738] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575a4ac561734433a144a2958a5a98e2, State: Initialized, Role: FOLLOWER
I20250115 02:53:46.658890 12061 consensus_queue.cc:260] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.665166 11789 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:46.668677 11800 leader_election.cc:304] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6d85313a5347e8bc7f4e91ead0c855, f546b9ebad5f44aa8afa2e3d62e3ca95; no voters: 575a4ac561734433a144a2958a5a98e2
I20250115 02:53:46.670828 12058 raft_consensus.cc:2798] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:46.671231 12058 raft_consensus.cc:491] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:46.671540 12058 raft_consensus.cc:3054] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:46.678545 12058 raft_consensus.cc:513] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.680675 12058 leader_election.cc:290] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 election: Requested vote from peers 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489), f546b9ebad5f44aa8afa2e3d62e3ca95 (127.2.36.67:37089)
I20250115 02:53:46.682549 12005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e9503376e842b790858102bb4d8f1b" candidate_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95"
I20250115 02:53:46.683307 12005 raft_consensus.cc:3054] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:46.685235 11771 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff: acquiring CA information for follower catalog manager: success
I20250115 02:53:46.682482 11927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e9503376e842b790858102bb4d8f1b" candidate_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a4ac561734433a144a2958a5a98e2"
I20250115 02:53:46.687145 11927 raft_consensus.cc:3054] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:46.690359 12005 raft_consensus.cc:2463] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6d85313a5347e8bc7f4e91ead0c855 in term 1.
I20250115 02:53:46.691761 11800 leader_election.cc:304] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6d85313a5347e8bc7f4e91ead0c855, f546b9ebad5f44aa8afa2e3d62e3ca95; no voters:
I20250115 02:53:46.692691 12058 raft_consensus.cc:2798] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:46.693220 12058 raft_consensus.cc:695] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 LEADER]: Becoming Leader. State: Replica: cf6d85313a5347e8bc7f4e91ead0c855, State: Running, Role: LEADER
I20250115 02:53:46.694056 12058 consensus_queue.cc:237] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.694793 12054 raft_consensus.cc:491] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:46.695513 12054 raft_consensus.cc:513] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } }
I20250115 02:53:46.697878 12054 leader_election.cc:290] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489), cf6d85313a5347e8bc7f4e91ead0c855 (127.2.36.65:33647)
I20250115 02:53:46.698513 12061 ts_tablet_manager.cc:1428] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2: Time spent starting tablet: real 0.061s user 0.011s sys 0.016s
I20250115 02:53:46.699540 11771 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:46.699640 12061 tablet_bootstrap.cc:492] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2: Bootstrap starting.
I20250115 02:53:46.705173 12061 tablet_bootstrap.cc:654] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:46.710280 11927 raft_consensus.cc:2463] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6d85313a5347e8bc7f4e91ead0c855 in term 1.
I20250115 02:53:46.725205 12061 tablet_bootstrap.cc:492] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2: No bootstrap required, opened a new log
I20250115 02:53:46.726507 12061 ts_tablet_manager.cc:1397] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.016s
I20250115 02:53:46.730345 11645 catalog_manager.cc:5526] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf6d85313a5347e8bc7f4e91ead0c855 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:46.731736 12061 raft_consensus.cc:357] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } }
I20250115 02:53:46.734200 12061 raft_consensus.cc:383] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:46.734951 12061 raft_consensus.cc:738] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575a4ac561734433a144a2958a5a98e2, State: Initialized, Role: FOLLOWER
I20250115 02:53:46.736021 12061 consensus_queue.cc:260] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } }
I20250115 02:53:46.740175 11927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fcbfa352bd4f0f9554a6e1d358e054" candidate_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a4ac561734433a144a2958a5a98e2" is_pre_election: true
I20250115 02:53:46.740836 11927 raft_consensus.cc:2463] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f546b9ebad5f44aa8afa2e3d62e3ca95 in term 0.
I20250115 02:53:46.741565 12061 ts_tablet_manager.cc:1428] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20250115 02:53:46.742659 12061 tablet_bootstrap.cc:492] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2: Bootstrap starting.
I20250115 02:53:46.744043 12058 raft_consensus.cc:491] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:46.744529 12058 raft_consensus.cc:513] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } }
I20250115 02:53:46.746451 12058 leader_election.cc:290] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f546b9ebad5f44aa8afa2e3d62e3ca95 (127.2.36.67:37089), 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489)
I20250115 02:53:46.747381 11968 leader_election.cc:304] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575a4ac561734433a144a2958a5a98e2, f546b9ebad5f44aa8afa2e3d62e3ca95; no voters:
I20250115 02:53:46.748095 11927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fcbfa352bd4f0f9554a6e1d358e054" candidate_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a4ac561734433a144a2958a5a98e2" is_pre_election: true
I20250115 02:53:46.748729 11927 raft_consensus.cc:2463] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6d85313a5347e8bc7f4e91ead0c855 in term 0.
I20250115 02:53:46.748096 12005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fcbfa352bd4f0f9554a6e1d358e054" candidate_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" is_pre_election: true
I20250115 02:53:46.749913 12061 tablet_bootstrap.cc:654] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:46.750099 12005 raft_consensus.cc:2463] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6d85313a5347e8bc7f4e91ead0c855 in term 0.
I20250115 02:53:46.752918 11800 leader_election.cc:304] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6d85313a5347e8bc7f4e91ead0c855, f546b9ebad5f44aa8afa2e3d62e3ca95; no voters:
I20250115 02:53:46.755393 12058 raft_consensus.cc:2798] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:46.755800 12058 raft_consensus.cc:491] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:46.755920 12054 raft_consensus.cc:2798] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:46.758706 12054 raft_consensus.cc:491] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:46.759196 12054 raft_consensus.cc:3054] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:46.756244 12058 raft_consensus.cc:3054] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:46.770521 11836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fcbfa352bd4f0f9554a6e1d358e054" candidate_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" is_pre_election: true
I20250115 02:53:46.778818 12061 tablet_bootstrap.cc:492] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2: No bootstrap required, opened a new log
I20250115 02:53:46.779523 12061 ts_tablet_manager.cc:1397] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.000s
I20250115 02:53:46.779263 12054 raft_consensus.cc:513] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } }
I20250115 02:53:46.781271 12054 leader_election.cc:290] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [CANDIDATE]: Term 1 election: Requested vote from peers 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489), cf6d85313a5347e8bc7f4e91ead0c855 (127.2.36.65:33647)
I20250115 02:53:46.782876 11927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fcbfa352bd4f0f9554a6e1d358e054" candidate_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a4ac561734433a144a2958a5a98e2"
I20250115 02:53:46.783509 11927 raft_consensus.cc:3054] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:46.783435 12061 raft_consensus.cc:357] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.784332 12061 raft_consensus.cc:383] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:46.784621 12061 raft_consensus.cc:738] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575a4ac561734433a144a2958a5a98e2, State: Initialized, Role: FOLLOWER
I20250115 02:53:46.784893 12058 raft_consensus.cc:513] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } }
I20250115 02:53:46.785308 12061 consensus_queue.cc:260] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.786098 11836 raft_consensus.cc:2388] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f546b9ebad5f44aa8afa2e3d62e3ca95 in current term 1: Already voted for candidate cf6d85313a5347e8bc7f4e91ead0c855 in this term.
I20250115 02:53:46.788262 12058 leader_election.cc:290] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 election: Requested vote from peers f546b9ebad5f44aa8afa2e3d62e3ca95 (127.2.36.67:37089), 575a4ac561734433a144a2958a5a98e2 (127.2.36.66:34489)
I20250115 02:53:46.788303 12005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fcbfa352bd4f0f9554a6e1d358e054" candidate_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95"
I20250115 02:53:46.789261 12005 raft_consensus.cc:2388] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6d85313a5347e8bc7f4e91ead0c855 in current term 1: Already voted for candidate f546b9ebad5f44aa8afa2e3d62e3ca95 in this term.
I20250115 02:53:46.789837 11926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fcbfa352bd4f0f9554a6e1d358e054" candidate_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a4ac561734433a144a2958a5a98e2"
I20250115 02:53:46.791438 11799 leader_election.cc:304] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6d85313a5347e8bc7f4e91ead0c855; no voters: 575a4ac561734433a144a2958a5a98e2, f546b9ebad5f44aa8afa2e3d62e3ca95
I20250115 02:53:46.788386 11835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fcbfa352bd4f0f9554a6e1d358e054" candidate_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf6d85313a5347e8bc7f4e91ead0c855"
I20250115 02:53:46.793171 12061 ts_tablet_manager.cc:1428] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2: Time spent starting tablet: real 0.013s user 0.003s sys 0.003s
I20250115 02:53:46.793449 11835 raft_consensus.cc:2388] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f546b9ebad5f44aa8afa2e3d62e3ca95 in current term 1: Already voted for candidate cf6d85313a5347e8bc7f4e91ead0c855 in this term.
I20250115 02:53:46.795120 12058 raft_consensus.cc:2743] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250115 02:53:46.803480 12068 raft_consensus.cc:491] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:46.804001 12068 raft_consensus.cc:513] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } }
I20250115 02:53:46.806839 11927 raft_consensus.cc:2463] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f546b9ebad5f44aa8afa2e3d62e3ca95 in term 1.
I20250115 02:53:46.807372 12068 leader_election.cc:290] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf6d85313a5347e8bc7f4e91ead0c855 (127.2.36.65:33647), f546b9ebad5f44aa8afa2e3d62e3ca95 (127.2.36.67:37089)
I20250115 02:53:46.819949 11968 leader_election.cc:304] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 575a4ac561734433a144a2958a5a98e2, f546b9ebad5f44aa8afa2e3d62e3ca95; no voters: cf6d85313a5347e8bc7f4e91ead0c855
I20250115 02:53:46.821189 12054 raft_consensus.cc:2798] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:46.824328 12054 raft_consensus.cc:695] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 LEADER]: Becoming Leader. State: Replica: f546b9ebad5f44aa8afa2e3d62e3ca95, State: Running, Role: LEADER
I20250115 02:53:46.825212 12054 consensus_queue.cc:237] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } } peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } }
I20250115 02:53:46.827874 12005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66df73085f843aeacb448f6a6036e19" candidate_uuid: "575a4ac561734433a144a2958a5a98e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" is_pre_election: true
I20250115 02:53:46.828711 12005 raft_consensus.cc:2388] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 575a4ac561734433a144a2958a5a98e2 in current term 1: Already voted for candidate cf6d85313a5347e8bc7f4e91ead0c855 in this term.
I20250115 02:53:46.832152 11836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66df73085f843aeacb448f6a6036e19" candidate_uuid: "575a4ac561734433a144a2958a5a98e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" is_pre_election: true
I20250115 02:53:46.834774 11890 leader_election.cc:304] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 575a4ac561734433a144a2958a5a98e2; no voters: cf6d85313a5347e8bc7f4e91ead0c855, f546b9ebad5f44aa8afa2e3d62e3ca95
I20250115 02:53:46.836124 12068 raft_consensus.cc:3054] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:46.842396 12068 raft_consensus.cc:2743] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:53:46.842917 11643 catalog_manager.cc:5526] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 reported cstate change: term changed from 0 to 1, leader changed from <none> to f546b9ebad5f44aa8afa2e3d62e3ca95 (127.2.36.67). New cstate: current_term: 1 leader_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:47.162775 12058 consensus_queue.cc:1035] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:47.251721 12095 consensus_queue.cc:1035] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:47.266079 12092 consensus_queue.cc:1035] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 02:53:47.302176 12096 consensus_queue.cc:1035] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f546b9ebad5f44aa8afa2e3d62e3ca95" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 37089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:47.363369 12054 consensus_queue.cc:1035] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "575a4ac561734433a144a2958a5a98e2" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:47.426931 12054 consensus_queue.cc:1035] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6d85313a5347e8bc7f4e91ead0c855" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 33647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:47.777654 11915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:53:47.806085 11993 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:53:47.813082 11826 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250115 02:53:48.204686 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:53:48.257490 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:48.258100 2193 tablet_replica.cc:331] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855: stopping tablet replica
I20250115 02:53:48.258711 2193 raft_consensus.cc:2238] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:48.259161 2193 raft_consensus.cc:2267] T 52fcbfa352bd4f0f9554a6e1d358e054 P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:48.261762 2193 tablet_replica.cc:331] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855: stopping tablet replica
I20250115 02:53:48.262497 2193 raft_consensus.cc:2238] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:48.263507 2193 raft_consensus.cc:2267] T c8e9503376e842b790858102bb4d8f1b P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:48.267176 2193 tablet_replica.cc:331] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855: stopping tablet replica
I20250115 02:53:48.267617 2193 raft_consensus.cc:2238] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:48.268319 2193 raft_consensus.cc:2267] T e66df73085f843aeacb448f6a6036e19 P cf6d85313a5347e8bc7f4e91ead0c855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:48.284370 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:53:48.300237 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:53:48.328248 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:48.328999 2193 tablet_replica.cc:331] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2: stopping tablet replica
I20250115 02:53:48.329725 2193 raft_consensus.cc:2238] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:48.330251 2193 raft_consensus.cc:2267] T e66df73085f843aeacb448f6a6036e19 P 575a4ac561734433a144a2958a5a98e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:48.332963 2193 tablet_replica.cc:331] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2: stopping tablet replica
I20250115 02:53:48.333509 2193 raft_consensus.cc:2238] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:48.333930 2193 raft_consensus.cc:2267] T 52fcbfa352bd4f0f9554a6e1d358e054 P 575a4ac561734433a144a2958a5a98e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:48.335987 2193 tablet_replica.cc:331] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2: stopping tablet replica
I20250115 02:53:48.336431 2193 raft_consensus.cc:2238] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:48.336767 2193 raft_consensus.cc:2267] T c8e9503376e842b790858102bb4d8f1b P 575a4ac561734433a144a2958a5a98e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:48.358249 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:53:48.370237 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:53:48.394107 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:48.394644 2193 tablet_replica.cc:331] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95: stopping tablet replica
I20250115 02:53:48.395212 2193 raft_consensus.cc:2238] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:48.395630 2193 raft_consensus.cc:2267] T c8e9503376e842b790858102bb4d8f1b P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:48.397425 2193 tablet_replica.cc:331] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95: stopping tablet replica
I20250115 02:53:48.397886 2193 raft_consensus.cc:2238] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:48.398253 2193 raft_consensus.cc:2267] T e66df73085f843aeacb448f6a6036e19 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:48.399933 2193 tablet_replica.cc:331] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95: stopping tablet replica
I20250115 02:53:48.400373 2193 raft_consensus.cc:2238] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:48.401043 2193 raft_consensus.cc:2267] T 52fcbfa352bd4f0f9554a6e1d358e054 P f546b9ebad5f44aa8afa2e3d62e3ca95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:48.411031 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:53:48.424434 2193 master.cc:537] Master@127.2.36.126:46643 shutting down...
I20250115 02:53:48.443084 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:48.443583 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:48.443878 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 800b9250ed1942dc98e9a9684a27dcff: stopping tablet replica
I20250115 02:53:48.462857 2193 master.cc:559] Master@127.2.36.126:46643 shutdown complete.
I20250115 02:53:48.473117 2193 master.cc:537] Master@127.2.36.125:39169 shutting down...
W20250115 02:53:48.495407 11629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.36.126:46643: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20250115 02:53:48.497391 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:48.498248 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:48.498639 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 153981cfa51f4b1ea656612d7e0e4a45: stopping tablet replica
I20250115 02:53:48.517887 2193 master.cc:559] Master@127.2.36.125:39169 shutdown complete.
I20250115 02:53:48.531028 2193 master.cc:537] Master@127.2.36.124:36523 shutting down...
I20250115 02:53:48.546408 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:48.546940 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:48.547226 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 8e67cb733f1a43b59140c0fcb8e1f9d7: stopping tablet replica
I20250115 02:53:48.565214 2193 master.cc:559] Master@127.2.36.124:36523 shutdown complete.
[ OK ] RemoteKsckTest.TestNonMatchingTabletIdFilter (3320 ms)
[ RUN ] RemoteKsckTest.TestMatchingTabletIdFilter
I20250115 02:53:48.591871 2193 test_util.cc:274] Using random seed: -915714327
I20250115 02:53:48.592785 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:33917,127.2.36.125:36725,127.2.36.124:33307
I20250115 02:53:48.593990 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:48.598876 12140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:48.598996 12141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:48.600078 12143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:48.601235 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:48.602061 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:48.602246 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:48.602393 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909628602376 us; error 0 us; skew 500 ppm
I20250115 02:53:48.603042 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:48.605254 2193 webserver.cc:458] Webserver started at http://127.2.36.126:33245/ using document root <none> and password file <none>
I20250115 02:53:48.605705 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:48.605870 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:48.606094 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:48.607148 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "f6493473251b4459b55d3110560296b4"
format_stamp: "Formatted at 2025-01-15 02:53:48 on dist-test-slave-1lv2"
I20250115 02:53:48.611156 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250115 02:53:48.614243 12148 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:48.614997 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:53:48.615247 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "f6493473251b4459b55d3110560296b4"
format_stamp: "Formatted at 2025-01-15 02:53:48 on dist-test-slave-1lv2"
I20250115 02:53:48.615495 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:48.640434 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:48.641369 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:48.677515 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:33917
I20250115 02:53:48.677603 12199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:33917 every 8 connection(s)
I20250115 02:53:48.681113 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:48.681205 12200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 02:53:48.686792 12202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:48.687260 12203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:48.688491 12205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:48.689774 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:48.689819 12200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:48.690701 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:48.690951 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:48.691151 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909628691131 us; error 0 us; skew 500 ppm
I20250115 02:53:48.691938 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:48.695003 2193 webserver.cc:458] Webserver started at http://127.2.36.125:35697/ using document root <none> and password file <none>
I20250115 02:53:48.695557 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:48.695793 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:48.696090 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:48.697451 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "52d0c80d24ce4c388bd368472469fcc1"
format_stamp: "Formatted at 2025-01-15 02:53:48 on dist-test-slave-1lv2"
I20250115 02:53:48.701879 12200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:48.703073 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
W20250115 02:53:48.707067 12200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:36725: Network error: Client connection negotiation failed: client connection to 127.2.36.125:36725: connect: Connection refused (error 111)
I20250115 02:53:48.707418 12213 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:48.708168 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s
I20250115 02:53:48.708436 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "52d0c80d24ce4c388bd368472469fcc1"
format_stamp: "Formatted at 2025-01-15 02:53:48 on dist-test-slave-1lv2"
I20250115 02:53:48.708694 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:48.722719 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:48.723707 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:48.756940 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:36725
I20250115 02:53:48.757022 12264 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:36725 every 8 connection(s)
I20250115 02:53:48.759573 12200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } attempt: 1
I20250115 02:53:48.760757 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:48.760793 12265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 02:53:48.766896 12268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:48.767966 12269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:48.770618 12265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:48.777604 12200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:48.778319 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:48.780979 12271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:48.782039 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:48.782344 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:48.782533 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909628782517 us; error 0 us; skew 500 ppm
W20250115 02:53:48.782802 12200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:33307: Network error: Client connection negotiation failed: client connection to 127.2.36.124:33307: connect: Connection refused (error 111)
I20250115 02:53:48.783237 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:48.785981 2193 webserver.cc:458] Webserver started at http://127.2.36.124:38853/ using document root <none> and password file <none>
I20250115 02:53:48.786513 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:48.786695 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:48.787024 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:48.788280 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "7eab4d6153c94c28b7ff84decb4fa22d"
format_stamp: "Formatted at 2025-01-15 02:53:48 on dist-test-slave-1lv2"
I20250115 02:53:48.788760 12265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:48.793973 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250115 02:53:48.798352 12277 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:48.799126 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:48.799288 12265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:48.799491 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "7eab4d6153c94c28b7ff84decb4fa22d"
format_stamp: "Formatted at 2025-01-15 02:53:48 on dist-test-slave-1lv2"
I20250115 02:53:48.799842 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250115 02:53:48.804639 12265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:33307: Network error: Client connection negotiation failed: client connection to 127.2.36.124:33307: connect: Connection refused (error 111)
I20250115 02:53:48.806253 12200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } attempt: 1
W20250115 02:53:48.809877 12200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:33307: Network error: Client connection negotiation failed: client connection to 127.2.36.124:33307: connect: Connection refused (error 111)
I20250115 02:53:48.828209 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:48.829249 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:48.833141 12265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } attempt: 1
W20250115 02:53:48.837076 12265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:33307: Network error: Client connection negotiation failed: client connection to 127.2.36.124:33307: connect: Connection refused (error 111)
I20250115 02:53:48.864555 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:33307
I20250115 02:53:48.864662 12329 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:33307 every 8 connection(s)
I20250115 02:53:48.867401 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:53:48.868088 12330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:48.871532 12265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } attempt: 2
I20250115 02:53:48.875021 12330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:48.885699 12330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:48.888469 12265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1: Bootstrap starting.
I20250115 02:53:48.891311 12200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } attempt: 2
I20250115 02:53:48.894508 12265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:48.896024 12330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:48.900112 12265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1: No bootstrap required, opened a new log
I20250115 02:53:48.903139 12265 raft_consensus.cc:357] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } }
I20250115 02:53:48.903700 12265 raft_consensus.cc:383] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:48.903934 12265 raft_consensus.cc:738] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d0c80d24ce4c388bd368472469fcc1, State: Initialized, Role: FOLLOWER
I20250115 02:53:48.904590 12265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } }
I20250115 02:53:48.906849 12336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } } }
I20250115 02:53:48.907720 12336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:48.908507 12265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:48.910028 12200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4: Bootstrap starting.
I20250115 02:53:48.915473 12200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:48.916380 12330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d: Bootstrap starting.
I20250115 02:53:48.922808 12330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:48.925963 12200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4: No bootstrap required, opened a new log
W20250115 02:53:48.926527 12348 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250115 02:53:48.926777 12330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d: No bootstrap required, opened a new log
W20250115 02:53:48.926949 12348 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:48.928273 12200 raft_consensus.cc:357] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } }
I20250115 02:53:48.928823 12200 raft_consensus.cc:383] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:48.929121 12200 raft_consensus.cc:738] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6493473251b4459b55d3110560296b4, State: Initialized, Role: FOLLOWER
I20250115 02:53:48.928876 12330 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } }
I20250115 02:53:48.929530 12330 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:48.929744 12330 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eab4d6153c94c28b7ff84decb4fa22d, State: Initialized, Role: FOLLOWER
I20250115 02:53:48.929769 12200 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } }
I20250115 02:53:48.930320 12330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } }
I20250115 02:53:48.931764 12350 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } } }
I20250115 02:53:48.932135 12351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } } }
I20250115 02:53:48.932617 12350 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:48.932988 12351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:48.933524 12200 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:48.933761 12330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:48.938944 12350 raft_consensus.cc:491] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:48.939411 12350 raft_consensus.cc:513] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } }
I20250115 02:53:48.941052 12350 leader_election.cc:290] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52d0c80d24ce4c388bd368472469fcc1 (127.2.36.125:36725), 7eab4d6153c94c28b7ff84decb4fa22d (127.2.36.124:33307)
I20250115 02:53:48.941773 12240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6493473251b4459b55d3110560296b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52d0c80d24ce4c388bd368472469fcc1" is_pre_election: true
I20250115 02:53:48.942526 12240 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6493473251b4459b55d3110560296b4 in term 0.
I20250115 02:53:48.943639 12305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6493473251b4459b55d3110560296b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" is_pre_election: true
I20250115 02:53:48.944383 12305 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6493473251b4459b55d3110560296b4 in term 0.
I20250115 02:53:48.945465 12151 leader_election.cc:304] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eab4d6153c94c28b7ff84decb4fa22d, f6493473251b4459b55d3110560296b4; no voters:
I20250115 02:53:48.946673 12350 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:48.947002 12350 raft_consensus.cc:491] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:48.947324 12350 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 0 FOLLOWER]: Advancing to term 1
W20250115 02:53:48.952661 12367 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:48.953094 12367 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:48.954387 12350 raft_consensus.cc:513] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } }
I20250115 02:53:48.956584 12350 leader_election.cc:290] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [CANDIDATE]: Term 1 election: Requested vote from peers 52d0c80d24ce4c388bd368472469fcc1 (127.2.36.125:36725), 7eab4d6153c94c28b7ff84decb4fa22d (127.2.36.124:33307)
I20250115 02:53:48.956820 12240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6493473251b4459b55d3110560296b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52d0c80d24ce4c388bd368472469fcc1"
I20250115 02:53:48.957144 12305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6493473251b4459b55d3110560296b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eab4d6153c94c28b7ff84decb4fa22d"
I20250115 02:53:48.957459 12240 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:48.957792 12305 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:48.960544 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:53:48.960821 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250115 02:53:48.960851 12373 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:48.961275 12373 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:48.962321 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:48.964382 12305 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6493473251b4459b55d3110560296b4 in term 1.
I20250115 02:53:48.964386 12240 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6493473251b4459b55d3110560296b4 in term 1.
I20250115 02:53:48.965575 12149 leader_election.cc:304] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52d0c80d24ce4c388bd368472469fcc1, f6493473251b4459b55d3110560296b4; no voters:
I20250115 02:53:48.966181 12350 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:48.968191 12350 raft_consensus.cc:695] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 1 LEADER]: Becoming Leader. State: Replica: f6493473251b4459b55d3110560296b4, State: Running, Role: LEADER
W20250115 02:53:48.968809 12374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:48.969426 12375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:48.969121 12350 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } }
I20250115 02:53:48.972261 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:48.972941 12378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:48.973889 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:48.974162 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:48.974423 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909628974402 us; error 0 us; skew 500 ppm
I20250115 02:53:48.974647 12376 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6493473251b4459b55d3110560296b4. Latest consensus state: current_term: 1 leader_uuid: "f6493473251b4459b55d3110560296b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } } }
I20250115 02:53:48.975235 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:48.975457 12376 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:48.976682 12381 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:48.978062 2193 webserver.cc:458] Webserver started at http://127.2.36.65:35065/ using document root <none> and password file <none>
I20250115 02:53:48.978560 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:48.978720 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:48.979002 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:48.979977 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "4889f8481be94852a147a28749d1e59d"
format_stamp: "Formatted at 2025-01-15 02:53:48 on dist-test-slave-1lv2"
I20250115 02:53:48.981562 12381 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:48.984401 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:53:48.989962 12305 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [term 1 FOLLOWER]: Refusing update from remote peer f6493473251b4459b55d3110560296b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:48.989951 12240 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [term 1 FOLLOWER]: Refusing update from remote peer f6493473251b4459b55d3110560296b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:48.990126 12386 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:48.991626 2193 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20250115 02:53:48.991729 12350 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:48.992017 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "4889f8481be94852a147a28749d1e59d"
format_stamp: "Formatted at 2025-01-15 02:53:48 on dist-test-slave-1lv2"
I20250115 02:53:48.992389 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:48.992461 12376 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:49.003479 12336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6493473251b4459b55d3110560296b4. Latest consensus state: current_term: 1 leader_uuid: "f6493473251b4459b55d3110560296b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } } }
I20250115 02:53:49.004352 12336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:49.008700 12351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6493473251b4459b55d3110560296b4. Latest consensus state: current_term: 1 leader_uuid: "f6493473251b4459b55d3110560296b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } } }
I20250115 02:53:49.009523 12351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:49.018489 12376 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6493473251b4459b55d3110560296b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } } }
I20250115 02:53:49.019374 12376 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:49.020148 12350 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6493473251b4459b55d3110560296b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } } }
I20250115 02:53:49.020965 12350 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:49.022148 12381 catalog_manager.cc:1348] Generated new cluster ID: 1f6e9462c90d4e2abfb80869028d1059
I20250115 02:53:49.022470 12381 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:49.026098 12336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6493473251b4459b55d3110560296b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } } }
I20250115 02:53:49.026890 12336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:49.027642 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:49.028733 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:49.029896 12351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6493473251b4459b55d3110560296b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6493473251b4459b55d3110560296b4" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33917 } } peers { permanent_uuid: "52d0c80d24ce4c388bd368472469fcc1" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 36725 } } peers { permanent_uuid: "7eab4d6153c94c28b7ff84decb4fa22d" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33307 } } }
I20250115 02:53:49.030711 12351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:49.033444 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:49.036065 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:49.036265 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:49.036517 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:49.036667 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:49.058688 12381 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:53:49.060473 12381 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:49.083273 12381 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4: Generated new TSK 0
I20250115 02:53:49.083963 12381 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:49.102046 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:38189
I20250115 02:53:49.102164 12456 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:38189 every 8 connection(s)
I20250115 02:53:49.128118 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:49.131649 12458 heartbeater.cc:346] Connected to a master server at 127.2.36.125:36725
I20250115 02:53:49.132054 12458 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:49.132316 12457 heartbeater.cc:346] Connected to a master server at 127.2.36.124:33307
I20250115 02:53:49.132748 12457 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:49.132871 12458 heartbeater.cc:509] Master 127.2.36.125:36725 requested a full tablet report, sending...
I20250115 02:53:49.133612 12457 heartbeater.cc:509] Master 127.2.36.124:33307 requested a full tablet report, sending...
I20250115 02:53:49.136030 12230 ts_manager.cc:194] Registered new tserver with Master: 4889f8481be94852a147a28749d1e59d (127.2.36.65:38189)
I20250115 02:53:49.136641 12295 ts_manager.cc:194] Registered new tserver with Master: 4889f8481be94852a147a28749d1e59d (127.2.36.65:38189)
W20250115 02:53:49.140853 12464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:49.146855 12459 heartbeater.cc:346] Connected to a master server at 127.2.36.126:33917
I20250115 02:53:49.147357 12459 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:49.148240 12459 heartbeater.cc:509] Master 127.2.36.126:33917 requested a full tablet report, sending...
I20250115 02:53:49.150835 12165 ts_manager.cc:194] Registered new tserver with Master: 4889f8481be94852a147a28749d1e59d (127.2.36.65:38189)
W20250115 02:53:49.151450 12467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:49.153309 12165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47614
I20250115 02:53:49.154052 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:49.155185 12465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:49.156203 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:49.156401 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:49.156536 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909629156526 us; error 0 us; skew 500 ppm
I20250115 02:53:49.156997 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:49.159260 2193 webserver.cc:458] Webserver started at http://127.2.36.66:45789/ using document root <none> and password file <none>
I20250115 02:53:49.159680 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:49.159832 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:49.160030 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:49.161006 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "1907ea7930484ac5bff2063c8349c6a1"
format_stamp: "Formatted at 2025-01-15 02:53:49 on dist-test-slave-1lv2"
I20250115 02:53:49.165817 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20250115 02:53:49.166136 12459 heartbeater.cc:501] Master 127.2.36.126:33917 was elected leader, sending a full tablet report...
I20250115 02:53:49.169616 12472 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:49.170420 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:53:49.170676 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "1907ea7930484ac5bff2063c8349c6a1"
format_stamp: "Formatted at 2025-01-15 02:53:49 on dist-test-slave-1lv2"
I20250115 02:53:49.170951 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:49.195641 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:49.196677 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:49.197935 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:49.200107 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:49.200310 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:49.200523 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:49.200704 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:49.244912 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:34921
I20250115 02:53:49.245002 12534 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:34921 every 8 connection(s)
I20250115 02:53:49.265245 12535 heartbeater.cc:346] Connected to a master server at 127.2.36.124:33307
I20250115 02:53:49.265674 12535 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:49.265865 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:49.266496 12535 heartbeater.cc:509] Master 127.2.36.124:33307 requested a full tablet report, sending...
I20250115 02:53:49.268492 12536 heartbeater.cc:346] Connected to a master server at 127.2.36.125:36725
I20250115 02:53:49.268860 12536 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:49.269210 12295 ts_manager.cc:194] Registered new tserver with Master: 1907ea7930484ac5bff2063c8349c6a1 (127.2.36.66:34921)
I20250115 02:53:49.269685 12536 heartbeater.cc:509] Master 127.2.36.125:36725 requested a full tablet report, sending...
I20250115 02:53:49.274958 12229 ts_manager.cc:194] Registered new tserver with Master: 1907ea7930484ac5bff2063c8349c6a1 (127.2.36.66:34921)
I20250115 02:53:49.279122 12537 heartbeater.cc:346] Connected to a master server at 127.2.36.126:33917
I20250115 02:53:49.279565 12537 heartbeater.cc:463] Registering TS with master...
W20250115 02:53:49.279937 12542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:49.280509 12537 heartbeater.cc:509] Master 127.2.36.126:33917 requested a full tablet report, sending...
W20250115 02:53:49.280692 12543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:49.281877 12545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:49.282778 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:49.282982 12165 ts_manager.cc:194] Registered new tserver with Master: 1907ea7930484ac5bff2063c8349c6a1 (127.2.36.66:34921)
I20250115 02:53:49.283679 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:49.283903 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:49.284071 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909629284056 us; error 0 us; skew 500 ppm
I20250115 02:53:49.284655 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:49.284859 12165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47626
I20250115 02:53:49.289682 2193 webserver.cc:458] Webserver started at http://127.2.36.67:45223/ using document root <none> and password file <none>
I20250115 02:53:49.290400 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:49.290614 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:49.290900 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:49.292136 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "01d572c1860b49bc86c802806fbe7530"
format_stamp: "Formatted at 2025-01-15 02:53:49 on dist-test-slave-1lv2"
I20250115 02:53:49.296475 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:53:49.297616 12537 heartbeater.cc:501] Master 127.2.36.126:33917 was elected leader, sending a full tablet report...
I20250115 02:53:49.300851 12550 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:49.302166 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250115 02:53:49.302541 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "01d572c1860b49bc86c802806fbe7530"
format_stamp: "Formatted at 2025-01-15 02:53:49 on dist-test-slave-1lv2"
I20250115 02:53:49.302918 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestMatchingTabletIdFilter.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:49.336505 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:49.337810 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:49.339459 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:49.341739 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:49.341998 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:49.342273 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:49.342473 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250115 02:53:49.399587 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:38337
I20250115 02:53:49.400650 12612 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:38337 every 8 connection(s)
I20250115 02:53:49.433648 12615 heartbeater.cc:346] Connected to a master server at 127.2.36.126:33917
I20250115 02:53:49.434084 12615 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:49.435021 12615 heartbeater.cc:509] Master 127.2.36.126:33917 requested a full tablet report, sending...
I20250115 02:53:49.437752 12613 heartbeater.cc:346] Connected to a master server at 127.2.36.124:33307
I20250115 02:53:49.437837 12164 ts_manager.cc:194] Registered new tserver with Master: 01d572c1860b49bc86c802806fbe7530 (127.2.36.67:38337)
I20250115 02:53:49.438184 12613 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:49.439426 12613 heartbeater.cc:509] Master 127.2.36.124:33307 requested a full tablet report, sending...
I20250115 02:53:49.439822 12164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47636
I20250115 02:53:49.442458 12614 heartbeater.cc:346] Connected to a master server at 127.2.36.125:36725
I20250115 02:53:49.445330 12614 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:49.445890 12614 heartbeater.cc:509] Master 127.2.36.125:36725 requested a full tablet report, sending...
I20250115 02:53:49.448287 12229 ts_manager.cc:194] Registered new tserver with Master: 01d572c1860b49bc86c802806fbe7530 (127.2.36.67:38337)
I20250115 02:53:49.448397 12294 ts_manager.cc:194] Registered new tserver with Master: 01d572c1860b49bc86c802806fbe7530 (127.2.36.67:38337)
I20250115 02:53:49.449178 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019605202s
I20250115 02:53:49.452639 12615 heartbeater.cc:501] Master 127.2.36.126:33917 was elected leader, sending a full tablet report...
I20250115 02:53:49.528894 12163 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47652:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:53:49.531304 12163 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:53:49.632272 12422 tablet_service.cc:1467] Processing CreateTablet for tablet f273a2492f444a1f902f659610081f23 (DEFAULT_TABLE table=ksck-test-table [id=3c7b0c0eb7664e0394f4ef61104d3cbe]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:49.633064 12578 tablet_service.cc:1467] Processing CreateTablet for tablet f273a2492f444a1f902f659610081f23 (DEFAULT_TABLE table=ksck-test-table [id=3c7b0c0eb7664e0394f4ef61104d3cbe]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:49.633661 12422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f273a2492f444a1f902f659610081f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:49.634335 12578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f273a2492f444a1f902f659610081f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:49.635542 12577 tablet_service.cc:1467] Processing CreateTablet for tablet 14a46142add34e52a5c62810bf5539ce (DEFAULT_TABLE table=ksck-test-table [id=3c7b0c0eb7664e0394f4ef61104d3cbe]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:49.636776 12577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14a46142add34e52a5c62810bf5539ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:49.632947 12576 tablet_service.cc:1467] Processing CreateTablet for tablet 90ba7cf46b20487497220a51cb63cb14 (DEFAULT_TABLE table=ksck-test-table [id=3c7b0c0eb7664e0394f4ef61104d3cbe]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:49.638046 12576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90ba7cf46b20487497220a51cb63cb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:49.646685 12421 tablet_service.cc:1467] Processing CreateTablet for tablet 14a46142add34e52a5c62810bf5539ce (DEFAULT_TABLE table=ksck-test-table [id=3c7b0c0eb7664e0394f4ef61104d3cbe]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:49.647652 12420 tablet_service.cc:1467] Processing CreateTablet for tablet 90ba7cf46b20487497220a51cb63cb14 (DEFAULT_TABLE table=ksck-test-table [id=3c7b0c0eb7664e0394f4ef61104d3cbe]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:49.648047 12421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14a46142add34e52a5c62810bf5539ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:49.653241 12500 tablet_service.cc:1467] Processing CreateTablet for tablet f273a2492f444a1f902f659610081f23 (DEFAULT_TABLE table=ksck-test-table [id=3c7b0c0eb7664e0394f4ef61104d3cbe]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:49.654577 12500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f273a2492f444a1f902f659610081f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:49.655733 12499 tablet_service.cc:1467] Processing CreateTablet for tablet 14a46142add34e52a5c62810bf5539ce (DEFAULT_TABLE table=ksck-test-table [id=3c7b0c0eb7664e0394f4ef61104d3cbe]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:49.667694 12499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14a46142add34e52a5c62810bf5539ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:49.658492 12498 tablet_service.cc:1467] Processing CreateTablet for tablet 90ba7cf46b20487497220a51cb63cb14 (DEFAULT_TABLE table=ksck-test-table [id=3c7b0c0eb7664e0394f4ef61104d3cbe]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:49.671769 12498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90ba7cf46b20487497220a51cb63cb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:49.660545 12420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90ba7cf46b20487497220a51cb63cb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:49.693049 12634 tablet_bootstrap.cc:492] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530: Bootstrap starting.
I20250115 02:53:49.694662 12635 tablet_bootstrap.cc:492] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d: Bootstrap starting.
I20250115 02:53:49.705067 12634 tablet_bootstrap.cc:654] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:49.711902 12635 tablet_bootstrap.cc:654] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:49.731818 12634 tablet_bootstrap.cc:492] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530: No bootstrap required, opened a new log
I20250115 02:53:49.735838 12634 ts_tablet_manager.cc:1397] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530: Time spent bootstrapping tablet: real 0.043s user 0.014s sys 0.004s
I20250115 02:53:49.743091 12638 tablet_bootstrap.cc:492] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1: Bootstrap starting.
I20250115 02:53:49.750290 12634 raft_consensus.cc:357] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } }
I20250115 02:53:49.751318 12634 raft_consensus.cc:383] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:49.751609 12634 raft_consensus.cc:738] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d572c1860b49bc86c802806fbe7530, State: Initialized, Role: FOLLOWER
I20250115 02:53:49.753088 12634 consensus_queue.cc:260] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } }
I20250115 02:53:49.758188 12635 tablet_bootstrap.cc:492] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d: No bootstrap required, opened a new log
I20250115 02:53:49.758829 12635 ts_tablet_manager.cc:1397] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.009s
I20250115 02:53:49.761543 12635 raft_consensus.cc:357] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:49.762228 12635 raft_consensus.cc:383] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:49.762523 12635 raft_consensus.cc:738] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4889f8481be94852a147a28749d1e59d, State: Initialized, Role: FOLLOWER
I20250115 02:53:49.763211 12635 consensus_queue.cc:260] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:49.767218 12638 tablet_bootstrap.cc:654] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:49.767261 12635 ts_tablet_manager.cc:1428] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20250115 02:53:49.782142 12635 tablet_bootstrap.cc:492] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d: Bootstrap starting.
I20250115 02:53:49.788236 12635 tablet_bootstrap.cc:654] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:49.792698 12634 ts_tablet_manager.cc:1428] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530: Time spent starting tablet: real 0.053s user 0.011s sys 0.022s
I20250115 02:53:49.795248 12634 tablet_bootstrap.cc:492] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530: Bootstrap starting.
I20250115 02:53:49.801234 12634 tablet_bootstrap.cc:654] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:49.816632 12635 tablet_bootstrap.cc:492] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d: No bootstrap required, opened a new log
I20250115 02:53:49.817334 12635 ts_tablet_manager.cc:1397] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.010s
I20250115 02:53:49.820187 12635 raft_consensus.cc:357] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:49.821059 12635 raft_consensus.cc:383] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:49.821424 12635 raft_consensus.cc:738] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4889f8481be94852a147a28749d1e59d, State: Initialized, Role: FOLLOWER
I20250115 02:53:49.822295 12635 consensus_queue.cc:260] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:49.829900 12638 tablet_bootstrap.cc:492] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1: No bootstrap required, opened a new log
I20250115 02:53:49.829957 12634 tablet_bootstrap.cc:492] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530: No bootstrap required, opened a new log
I20250115 02:53:49.830443 12638 ts_tablet_manager.cc:1397] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1: Time spent bootstrapping tablet: real 0.088s user 0.013s sys 0.007s
I20250115 02:53:49.830621 12634 ts_tablet_manager.cc:1397] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.024s
I20250115 02:53:49.833227 12638 raft_consensus.cc:357] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:49.833842 12635 ts_tablet_manager.cc:1428] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d: Time spent starting tablet: real 0.016s user 0.003s sys 0.003s
I20250115 02:53:49.834149 12638 raft_consensus.cc:383] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:49.834542 12638 raft_consensus.cc:738] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1907ea7930484ac5bff2063c8349c6a1, State: Initialized, Role: FOLLOWER
I20250115 02:53:49.835471 12638 consensus_queue.cc:260] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:49.841840 12634 raft_consensus.cc:357] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:49.844918 12634 raft_consensus.cc:383] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:49.845288 12634 raft_consensus.cc:738] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d572c1860b49bc86c802806fbe7530, State: Initialized, Role: FOLLOWER
I20250115 02:53:49.845721 12638 ts_tablet_manager.cc:1428] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20250115 02:53:49.846025 12634 consensus_queue.cc:260] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:49.852461 12634 ts_tablet_manager.cc:1428] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20250115 02:53:49.853565 12634 tablet_bootstrap.cc:492] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530: Bootstrap starting.
I20250115 02:53:49.855823 12638 tablet_bootstrap.cc:492] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1: Bootstrap starting.
I20250115 02:53:49.861938 12638 tablet_bootstrap.cc:654] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:49.869890 12634 tablet_bootstrap.cc:654] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:49.870652 12635 tablet_bootstrap.cc:492] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d: Bootstrap starting.
I20250115 02:53:49.880774 12635 tablet_bootstrap.cc:654] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:49.883211 12638 tablet_bootstrap.cc:492] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1: No bootstrap required, opened a new log
I20250115 02:53:49.883826 12638 ts_tablet_manager.cc:1397] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.005s
I20250115 02:53:49.884023 12640 raft_consensus.cc:491] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:49.884505 12640 raft_consensus.cc:513] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:49.890872 12640 leader_election.cc:290] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1907ea7930484ac5bff2063c8349c6a1 (127.2.36.66:34921), 01d572c1860b49bc86c802806fbe7530 (127.2.36.67:38337)
I20250115 02:53:49.886670 12638 raft_consensus.cc:357] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } }
I20250115 02:53:49.892056 12638 raft_consensus.cc:383] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:49.892405 12638 raft_consensus.cc:738] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1907ea7930484ac5bff2063c8349c6a1, State: Initialized, Role: FOLLOWER
I20250115 02:53:49.896919 12638 consensus_queue.cc:260] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } }
I20250115 02:53:49.901357 12638 ts_tablet_manager.cc:1428] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1: Time spent starting tablet: real 0.017s user 0.004s sys 0.003s
I20250115 02:53:49.915133 12638 tablet_bootstrap.cc:492] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1: Bootstrap starting.
I20250115 02:53:49.929253 12638 tablet_bootstrap.cc:654] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:49.948599 12348 catalog_manager.cc:1260] Loaded cluster ID: 1f6e9462c90d4e2abfb80869028d1059
I20250115 02:53:49.949290 12348 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1: loading cluster ID for follower catalog manager: success
I20250115 02:53:49.965942 12367 catalog_manager.cc:1260] Loaded cluster ID: 1f6e9462c90d4e2abfb80869028d1059
I20250115 02:53:49.966454 12367 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d: loading cluster ID for follower catalog manager: success
I20250115 02:53:49.974990 12367 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d: acquiring CA information for follower catalog manager: success
I20250115 02:53:50.014281 12510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a46142add34e52a5c62810bf5539ce" candidate_uuid: "4889f8481be94852a147a28749d1e59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1907ea7930484ac5bff2063c8349c6a1" is_pre_election: true
I20250115 02:53:50.003522 12634 tablet_bootstrap.cc:492] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530: No bootstrap required, opened a new log
I20250115 02:53:50.015120 12510 raft_consensus.cc:2463] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4889f8481be94852a147a28749d1e59d in term 0.
I20250115 02:53:50.015569 12634 ts_tablet_manager.cc:1397] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530: Time spent bootstrapping tablet: real 0.162s user 0.046s sys 0.057s
I20250115 02:53:50.018517 12393 leader_election.cc:304] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1907ea7930484ac5bff2063c8349c6a1, 4889f8481be94852a147a28749d1e59d; no voters:
I20250115 02:53:50.019526 12640 raft_consensus.cc:2798] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:50.019887 12640 raft_consensus.cc:491] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:50.020284 12367 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:50.020959 12588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a46142add34e52a5c62810bf5539ce" candidate_uuid: "4889f8481be94852a147a28749d1e59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d572c1860b49bc86c802806fbe7530" is_pre_election: true
I20250115 02:53:50.022238 12634 raft_consensus.cc:357] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:50.022991 12634 raft_consensus.cc:383] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:50.023288 12634 raft_consensus.cc:738] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d572c1860b49bc86c802806fbe7530, State: Initialized, Role: FOLLOWER
I20250115 02:53:50.023918 12634 consensus_queue.cc:260] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:50.020228 12640 raft_consensus.cc:3054] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:50.026698 12634 ts_tablet_manager.cc:1428] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250115 02:53:50.028831 12635 tablet_bootstrap.cc:492] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d: No bootstrap required, opened a new log
I20250115 02:53:50.029371 12638 tablet_bootstrap.cc:492] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1: No bootstrap required, opened a new log
I20250115 02:53:50.029389 12635 ts_tablet_manager.cc:1397] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d: Time spent bootstrapping tablet: real 0.164s user 0.004s sys 0.016s
I20250115 02:53:50.029865 12638 ts_tablet_manager.cc:1397] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1: Time spent bootstrapping tablet: real 0.115s user 0.022s sys 0.036s
W20250115 02:53:50.030424 12388 leader_election.cc:343] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 01d572c1860b49bc86c802806fbe7530 (127.2.36.67:38337): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:50.032905 12638 raft_consensus.cc:357] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:50.033696 12638 raft_consensus.cc:383] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:50.033994 12638 raft_consensus.cc:738] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1907ea7930484ac5bff2063c8349c6a1, State: Initialized, Role: FOLLOWER
I20250115 02:53:50.034689 12638 consensus_queue.cc:260] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:50.035676 12640 raft_consensus.cc:513] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:50.037123 12638 ts_tablet_manager.cc:1428] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250115 02:53:50.038301 12510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a46142add34e52a5c62810bf5539ce" candidate_uuid: "4889f8481be94852a147a28749d1e59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1907ea7930484ac5bff2063c8349c6a1"
I20250115 02:53:50.038905 12510 raft_consensus.cc:3054] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:50.046039 12510 raft_consensus.cc:2463] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4889f8481be94852a147a28749d1e59d in term 1.
I20250115 02:53:50.047091 12393 leader_election.cc:304] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1907ea7930484ac5bff2063c8349c6a1, 4889f8481be94852a147a28749d1e59d; no voters:
I20250115 02:53:50.046893 12635 raft_consensus.cc:357] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } }
I20250115 02:53:50.047832 12635 raft_consensus.cc:383] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:50.048246 12635 raft_consensus.cc:738] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4889f8481be94852a147a28749d1e59d, State: Initialized, Role: FOLLOWER
I20250115 02:53:50.048921 12635 consensus_queue.cc:260] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } }
I20250115 02:53:50.031263 12348 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1: acquiring CA information for follower catalog manager: success
W20250115 02:53:50.151955 12393 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.36.66:34921, user_credentials={real_user=slave}} blocked reactor thread for 104962us
I20250115 02:53:50.051486 12640 leader_election.cc:290] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 election: Requested vote from peers 1907ea7930484ac5bff2063c8349c6a1 (127.2.36.66:34921), 01d572c1860b49bc86c802806fbe7530 (127.2.36.67:38337)
I20250115 02:53:50.053999 12639 raft_consensus.cc:491] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:50.154633 12588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a46142add34e52a5c62810bf5539ce" candidate_uuid: "4889f8481be94852a147a28749d1e59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d572c1860b49bc86c802806fbe7530"
I20250115 02:53:50.151347 12652 raft_consensus.cc:2798] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:50.155604 12652 raft_consensus.cc:695] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 1 LEADER]: Becoming Leader. State: Replica: 4889f8481be94852a147a28749d1e59d, State: Running, Role: LEADER
I20250115 02:53:50.156519 12652 consensus_queue.cc:237] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:50.155321 12588 raft_consensus.cc:3054] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:50.165896 12348 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:50.152225 12651 raft_consensus.cc:491] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:50.168583 12651 raft_consensus.cc:513] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:50.169108 12588 raft_consensus.cc:2463] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4889f8481be94852a147a28749d1e59d in term 1.
I20250115 02:53:50.171406 12510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ba7cf46b20487497220a51cb63cb14" candidate_uuid: "4889f8481be94852a147a28749d1e59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1907ea7930484ac5bff2063c8349c6a1" is_pre_election: true
I20250115 02:53:50.172044 12510 raft_consensus.cc:2463] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4889f8481be94852a147a28749d1e59d in term 0.
I20250115 02:53:50.173077 12640 raft_consensus.cc:491] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:50.173811 12644 raft_consensus.cc:491] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:50.174276 12644 raft_consensus.cc:513] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } }
I20250115 02:53:50.176765 12644 leader_election.cc:290] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01d572c1860b49bc86c802806fbe7530 (127.2.36.67:38337), 4889f8481be94852a147a28749d1e59d (127.2.36.65:38189)
I20250115 02:53:50.178400 12651 leader_election.cc:290] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1907ea7930484ac5bff2063c8349c6a1 (127.2.36.66:34921), 01d572c1860b49bc86c802806fbe7530 (127.2.36.67:38337)
I20250115 02:53:50.173556 12640 raft_consensus.cc:513] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } }
I20250115 02:53:50.183054 12640 leader_election.cc:290] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1907ea7930484ac5bff2063c8349c6a1 (127.2.36.66:34921), 01d572c1860b49bc86c802806fbe7530 (127.2.36.67:38337)
I20250115 02:53:50.154222 12639 raft_consensus.cc:513] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:50.189110 12393 leader_election.cc:304] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1907ea7930484ac5bff2063c8349c6a1, 4889f8481be94852a147a28749d1e59d; no voters:
I20250115 02:53:50.188050 12510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273a2492f444a1f902f659610081f23" candidate_uuid: "4889f8481be94852a147a28749d1e59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1907ea7930484ac5bff2063c8349c6a1" is_pre_election: true
I20250115 02:53:50.189898 12510 raft_consensus.cc:2463] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4889f8481be94852a147a28749d1e59d in term 0.
I20250115 02:53:50.191159 12640 raft_consensus.cc:2798] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:50.191576 12640 raft_consensus.cc:491] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:50.191910 12640 raft_consensus.cc:3054] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:50.199604 12639 leader_election.cc:290] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4889f8481be94852a147a28749d1e59d (127.2.36.65:38189), 1907ea7930484ac5bff2063c8349c6a1 (127.2.36.66:34921)
I20250115 02:53:50.200325 12639 raft_consensus.cc:491] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:50.200872 12639 raft_consensus.cc:513] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } }
I20250115 02:53:50.204963 12393 leader_election.cc:304] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1907ea7930484ac5bff2063c8349c6a1, 4889f8481be94852a147a28749d1e59d; no voters:
I20250115 02:53:50.222236 12640 raft_consensus.cc:513] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:50.229053 12640 leader_election.cc:290] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 election: Requested vote from peers 1907ea7930484ac5bff2063c8349c6a1 (127.2.36.66:34921), 01d572c1860b49bc86c802806fbe7530 (127.2.36.67:38337)
I20250115 02:53:50.220453 12652 raft_consensus.cc:2798] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:50.226586 12588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ba7cf46b20487497220a51cb63cb14" candidate_uuid: "4889f8481be94852a147a28749d1e59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d572c1860b49bc86c802806fbe7530" is_pre_election: true
I20250115 02:53:50.226886 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273a2492f444a1f902f659610081f23" candidate_uuid: "4889f8481be94852a147a28749d1e59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d572c1860b49bc86c802806fbe7530" is_pre_election: true
I20250115 02:53:50.227216 12639 leader_election.cc:290] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1907ea7930484ac5bff2063c8349c6a1 (127.2.36.66:34921), 4889f8481be94852a147a28749d1e59d (127.2.36.65:38189)
I20250115 02:53:50.228686 12635 ts_tablet_manager.cc:1428] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d: Time spent starting tablet: real 0.197s user 0.019s sys 0.028s
I20250115 02:53:50.232756 12587 raft_consensus.cc:2463] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4889f8481be94852a147a28749d1e59d in term 0.
I20250115 02:53:50.230587 12588 raft_consensus.cc:2463] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4889f8481be94852a147a28749d1e59d in term 0.
I20250115 02:53:50.229799 12510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ba7cf46b20487497220a51cb63cb14" candidate_uuid: "4889f8481be94852a147a28749d1e59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1907ea7930484ac5bff2063c8349c6a1"
I20250115 02:53:50.235136 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273a2492f444a1f902f659610081f23" candidate_uuid: "1907ea7930484ac5bff2063c8349c6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d572c1860b49bc86c802806fbe7530" is_pre_election: true
I20250115 02:53:50.235816 12587 raft_consensus.cc:2463] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1907ea7930484ac5bff2063c8349c6a1 in term 0.
I20250115 02:53:50.233731 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ba7cf46b20487497220a51cb63cb14" candidate_uuid: "4889f8481be94852a147a28749d1e59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d572c1860b49bc86c802806fbe7530"
I20250115 02:53:50.236711 12586 raft_consensus.cc:3054] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:50.237973 12474 leader_election.cc:304] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d572c1860b49bc86c802806fbe7530, 1907ea7930484ac5bff2063c8349c6a1; no voters:
I20250115 02:53:50.238934 12644 raft_consensus.cc:2798] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:50.239332 12644 raft_consensus.cc:491] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:50.239662 12644 raft_consensus.cc:3054] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:50.235065 12510 raft_consensus.cc:3054] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:50.248423 12644 raft_consensus.cc:513] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } }
I20250115 02:53:50.251030 12586 raft_consensus.cc:2463] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4889f8481be94852a147a28749d1e59d in term 1.
I20250115 02:53:50.251652 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273a2492f444a1f902f659610081f23" candidate_uuid: "1907ea7930484ac5bff2063c8349c6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d572c1860b49bc86c802806fbe7530"
I20250115 02:53:50.252318 12587 raft_consensus.cc:3054] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:50.255865 12432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273a2492f444a1f902f659610081f23" candidate_uuid: "1907ea7930484ac5bff2063c8349c6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4889f8481be94852a147a28749d1e59d" is_pre_election: true
I20250115 02:53:50.262209 12587 raft_consensus.cc:2463] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1907ea7930484ac5bff2063c8349c6a1 in term 1.
I20250115 02:53:50.234144 12652 raft_consensus.cc:491] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:50.263989 12652 raft_consensus.cc:3054] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:50.269774 12644 leader_election.cc:290] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [CANDIDATE]: Term 1 election: Requested vote from peers 01d572c1860b49bc86c802806fbe7530 (127.2.36.67:38337), 4889f8481be94852a147a28749d1e59d (127.2.36.65:38189)
I20250115 02:53:50.271150 12652 raft_consensus.cc:513] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } }
I20250115 02:53:50.272728 12652 leader_election.cc:290] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 election: Requested vote from peers 1907ea7930484ac5bff2063c8349c6a1 (127.2.36.66:34921), 01d572c1860b49bc86c802806fbe7530 (127.2.36.67:38337)
I20250115 02:53:50.313117 12474 leader_election.cc:304] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d572c1860b49bc86c802806fbe7530, 1907ea7930484ac5bff2063c8349c6a1; no voters:
I20250115 02:53:50.314730 12432 raft_consensus.cc:2388] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1907ea7930484ac5bff2063c8349c6a1 in current term 1: Already voted for candidate 4889f8481be94852a147a28749d1e59d in this term.
I20250115 02:53:50.254328 12510 raft_consensus.cc:2463] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4889f8481be94852a147a28749d1e59d in term 1.
I20250115 02:53:50.319854 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273a2492f444a1f902f659610081f23" candidate_uuid: "4889f8481be94852a147a28749d1e59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d572c1860b49bc86c802806fbe7530"
I20250115 02:53:50.321707 12587 raft_consensus.cc:2388] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4889f8481be94852a147a28749d1e59d in current term 1: Already voted for candidate 1907ea7930484ac5bff2063c8349c6a1 in this term.
I20250115 02:53:50.322093 12388 leader_election.cc:304] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d572c1860b49bc86c802806fbe7530, 4889f8481be94852a147a28749d1e59d; no voters:
I20250115 02:53:50.323585 12431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273a2492f444a1f902f659610081f23" candidate_uuid: "1907ea7930484ac5bff2063c8349c6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4889f8481be94852a147a28749d1e59d"
I20250115 02:53:50.319943 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273a2492f444a1f902f659610081f23" candidate_uuid: "4889f8481be94852a147a28749d1e59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1907ea7930484ac5bff2063c8349c6a1"
I20250115 02:53:50.327533 12509 raft_consensus.cc:2388] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4889f8481be94852a147a28749d1e59d in current term 1: Already voted for candidate 1907ea7930484ac5bff2063c8349c6a1 in this term.
I20250115 02:53:50.328054 12652 raft_consensus.cc:2798] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:50.330139 12644 raft_consensus.cc:2798] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:50.331964 12393 leader_election.cc:304] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4889f8481be94852a147a28749d1e59d; no voters: 01d572c1860b49bc86c802806fbe7530, 1907ea7930484ac5bff2063c8349c6a1
I20250115 02:53:50.338745 12431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ba7cf46b20487497220a51cb63cb14" candidate_uuid: "01d572c1860b49bc86c802806fbe7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4889f8481be94852a147a28749d1e59d" is_pre_election: true
I20250115 02:53:50.333887 12652 raft_consensus.cc:695] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 1 LEADER]: Becoming Leader. State: Replica: 4889f8481be94852a147a28749d1e59d, State: Running, Role: LEADER
I20250115 02:53:50.340345 12640 raft_consensus.cc:2743] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250115 02:53:50.341252 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273a2492f444a1f902f659610081f23" candidate_uuid: "01d572c1860b49bc86c802806fbe7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4889f8481be94852a147a28749d1e59d" is_pre_election: true
I20250115 02:53:50.342995 12652 consensus_queue.cc:237] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } }
I20250115 02:53:50.343545 12165 catalog_manager.cc:5526] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4889f8481be94852a147a28749d1e59d (127.2.36.65). New cstate: current_term: 1 leader_uuid: "4889f8481be94852a147a28749d1e59d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:50.357045 12644 raft_consensus.cc:695] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 1 LEADER]: Becoming Leader. State: Replica: 1907ea7930484ac5bff2063c8349c6a1, State: Running, Role: LEADER
I20250115 02:53:50.364997 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ba7cf46b20487497220a51cb63cb14" candidate_uuid: "01d572c1860b49bc86c802806fbe7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1907ea7930484ac5bff2063c8349c6a1" is_pre_election: true
I20250115 02:53:50.365991 12509 raft_consensus.cc:2388] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01d572c1860b49bc86c802806fbe7530 in current term 1: Already voted for candidate 4889f8481be94852a147a28749d1e59d in this term.
I20250115 02:53:50.365612 12644 consensus_queue.cc:237] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } }
I20250115 02:53:50.367129 12510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273a2492f444a1f902f659610081f23" candidate_uuid: "01d572c1860b49bc86c802806fbe7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1907ea7930484ac5bff2063c8349c6a1" is_pre_election: true
I20250115 02:53:50.369940 12554 leader_election.cc:304] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01d572c1860b49bc86c802806fbe7530; no voters: 1907ea7930484ac5bff2063c8349c6a1, 4889f8481be94852a147a28749d1e59d
I20250115 02:53:50.372119 12639 raft_consensus.cc:2743] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:53:50.384588 12554 leader_election.cc:304] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01d572c1860b49bc86c802806fbe7530; no voters: 1907ea7930484ac5bff2063c8349c6a1, 4889f8481be94852a147a28749d1e59d
I20250115 02:53:50.385702 12639 raft_consensus.cc:2743] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:53:50.432077 12164 catalog_manager.cc:5526] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1907ea7930484ac5bff2063c8349c6a1 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "1907ea7930484ac5bff2063c8349c6a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:50.459901 12165 catalog_manager.cc:5526] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4889f8481be94852a147a28749d1e59d (127.2.36.65). New cstate: current_term: 1 leader_uuid: "4889f8481be94852a147a28749d1e59d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:50.619062 2193 ksck_remote-test.cc:739] Generating row writes...
I20250115 02:53:50.624078 12652 consensus_queue.cc:1035] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:50.680353 12652 consensus_queue.cc:1035] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:50.831102 12644 consensus_queue.cc:1035] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:50.836076 12644 consensus_queue.cc:1035] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4889f8481be94852a147a28749d1e59d" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:50.848075 12652 consensus_queue.cc:1035] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d572c1860b49bc86c802806fbe7530" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 38337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:50.905418 12652 consensus_queue.cc:1035] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1907ea7930484ac5bff2063c8349c6a1" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:51.926173 12498 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:53:51.931938 12577 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:53:51.946822 12422 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250115 02:53:52.377619 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:53:52.442550 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:52.443269 2193 tablet_replica.cc:331] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d: stopping tablet replica
I20250115 02:53:52.443853 2193 raft_consensus.cc:2238] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:52.444603 2193 raft_consensus.cc:2267] T 90ba7cf46b20487497220a51cb63cb14 P 4889f8481be94852a147a28749d1e59d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:52.447571 2193 tablet_replica.cc:331] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d: stopping tablet replica
I20250115 02:53:52.449463 2193 raft_consensus.cc:2238] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:52.450115 2193 raft_consensus.cc:2267] T f273a2492f444a1f902f659610081f23 P 4889f8481be94852a147a28749d1e59d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:52.454892 2193 tablet_replica.cc:331] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d: stopping tablet replica
I20250115 02:53:52.455674 2193 raft_consensus.cc:2238] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:52.456508 2193 raft_consensus.cc:2267] T 14a46142add34e52a5c62810bf5539ce P 4889f8481be94852a147a28749d1e59d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:52.484221 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:53:52.502596 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
W20250115 02:53:52.511827 12476 consensus_peers.cc:487] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 -> Peer 4889f8481be94852a147a28749d1e59d (127.2.36.65:38189): Couldn't send request to peer 4889f8481be94852a147a28749d1e59d. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:38189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:53:52.561283 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:52.561969 2193 tablet_replica.cc:331] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1: stopping tablet replica
I20250115 02:53:52.562716 2193 raft_consensus.cc:2238] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:52.563333 2193 raft_consensus.cc:2267] T 90ba7cf46b20487497220a51cb63cb14 P 1907ea7930484ac5bff2063c8349c6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:52.565853 2193 tablet_replica.cc:331] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1: stopping tablet replica
I20250115 02:53:52.566335 2193 raft_consensus.cc:2238] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:52.567093 2193 raft_consensus.cc:2267] T f273a2492f444a1f902f659610081f23 P 1907ea7930484ac5bff2063c8349c6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:52.569020 2193 tablet_replica.cc:331] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1: stopping tablet replica
I20250115 02:53:52.569509 2193 raft_consensus.cc:2238] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:52.570050 2193 raft_consensus.cc:2267] T 14a46142add34e52a5c62810bf5539ce P 1907ea7930484ac5bff2063c8349c6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:52.591775 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:53:52.612535 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:53:52.643733 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:52.644407 2193 tablet_replica.cc:331] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530: stopping tablet replica
I20250115 02:53:52.645009 2193 raft_consensus.cc:2238] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:52.645452 2193 raft_consensus.cc:2267] T 90ba7cf46b20487497220a51cb63cb14 P 01d572c1860b49bc86c802806fbe7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:52.647670 2193 tablet_replica.cc:331] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530: stopping tablet replica
I20250115 02:53:52.648155 2193 raft_consensus.cc:2238] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:52.648573 2193 raft_consensus.cc:2267] T 14a46142add34e52a5c62810bf5539ce P 01d572c1860b49bc86c802806fbe7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:52.650432 2193 tablet_replica.cc:331] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530: stopping tablet replica
I20250115 02:53:52.650880 2193 raft_consensus.cc:2238] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:52.651288 2193 raft_consensus.cc:2267] T f273a2492f444a1f902f659610081f23 P 01d572c1860b49bc86c802806fbe7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:52.672014 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:53:52.685231 2193 master.cc:537] Master@127.2.36.126:33917 shutting down...
I20250115 02:53:52.708575 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:52.709437 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:52.709786 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P f6493473251b4459b55d3110560296b4: stopping tablet replica
I20250115 02:53:52.732323 2193 master.cc:559] Master@127.2.36.126:33917 shutdown complete.
I20250115 02:53:52.747262 2193 master.cc:537] Master@127.2.36.125:36725 shutting down...
I20250115 02:53:52.765009 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:52.765514 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:52.765794 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 52d0c80d24ce4c388bd368472469fcc1: stopping tablet replica
I20250115 02:53:52.785647 2193 master.cc:559] Master@127.2.36.125:36725 shutdown complete.
I20250115 02:53:52.800117 2193 master.cc:537] Master@127.2.36.124:33307 shutting down...
I20250115 02:53:52.818591 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:52.819120 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:52.819407 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7eab4d6153c94c28b7ff84decb4fa22d: stopping tablet replica
I20250115 02:53:52.838048 2193 master.cc:559] Master@127.2.36.124:33307 shutdown complete.
[ OK ] RemoteKsckTest.TestMatchingTabletIdFilter (4278 ms)
[ RUN ] RemoteKsckTest.TestTableFiltersNoMatch
I20250115 02:53:52.870074 2193 test_util.cc:274] Using random seed: -911436122
I20250115 02:53:52.871062 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:35607,127.2.36.125:43219,127.2.36.124:44903
I20250115 02:53:52.872326 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:52.877445 12738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:52.877964 12739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:52.878649 12741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:52.879987 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:52.881492 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:52.881685 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:52.881814 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909632881799 us; error 0 us; skew 500 ppm
I20250115 02:53:52.882290 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:52.887792 2193 webserver.cc:458] Webserver started at http://127.2.36.126:37953/ using document root <none> and password file <none>
I20250115 02:53:52.888231 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:52.888394 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:52.888622 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:52.889721 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "552ffd497bfd4e3b9bda39b14d1610a6"
format_stamp: "Formatted at 2025-01-15 02:53:52 on dist-test-slave-1lv2"
I20250115 02:53:52.893807 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250115 02:53:52.896972 12746 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:52.897732 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:53:52.897979 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "552ffd497bfd4e3b9bda39b14d1610a6"
format_stamp: "Formatted at 2025-01-15 02:53:52 on dist-test-slave-1lv2"
I20250115 02:53:52.898212 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:52.909229 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:52.910331 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:52.946049 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:35607
I20250115 02:53:52.946146 12797 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:35607 every 8 connection(s)
I20250115 02:53:52.949744 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:52.949803 12798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 02:53:52.955144 12800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:52.956821 12801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:52.957939 12798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:52.958472 12803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:52.958858 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:52.960770 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:52.961087 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:52.961354 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909632961328 us; error 0 us; skew 500 ppm
I20250115 02:53:52.961969 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:52.964854 2193 webserver.cc:458] Webserver started at http://127.2.36.125:41763/ using document root <none> and password file <none>
I20250115 02:53:52.965469 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:52.965705 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:52.966003 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:52.967458 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "8dc82e9e96d04610b32c8a12c6973b0c"
format_stamp: "Formatted at 2025-01-15 02:53:52 on dist-test-slave-1lv2"
I20250115 02:53:52.970582 12798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:52.972316 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
W20250115 02:53:52.975833 12798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:43219: Network error: Client connection negotiation failed: client connection to 127.2.36.125:43219: connect: Connection refused (error 111)
I20250115 02:53:52.975975 12811 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:52.976785 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:52.977048 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "8dc82e9e96d04610b32c8a12c6973b0c"
format_stamp: "Formatted at 2025-01-15 02:53:52 on dist-test-slave-1lv2"
I20250115 02:53:52.977306 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:52.998318 12798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } attempt: 1
W20250115 02:53:53.002596 12798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:43219: Network error: Client connection negotiation failed: client connection to 127.2.36.125:43219: connect: Connection refused (error 111)
I20250115 02:53:53.003986 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:53.005223 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:53.043062 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:43219
I20250115 02:53:53.043151 12862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:43219 every 8 connection(s)
I20250115 02:53:53.046838 12863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:53.046977 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:53.051203 12798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } attempt: 2
W20250115 02:53:53.052474 12865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:53.053892 12866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:53.058199 12863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:53.059918 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:53.062908 12869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:53.063912 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:53.064149 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:53.064381 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909633064362 us; error 0 us; skew 500 ppm
I20250115 02:53:53.065220 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:53.068806 12798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:53.071041 2193 webserver.cc:458] Webserver started at http://127.2.36.124:42413/ using document root <none> and password file <none>
I20250115 02:53:53.071719 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:53.071985 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:53.072094 12863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:53.072301 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:53.073830 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "e800fe9800dc4ac8808518c32ff6c7aa"
format_stamp: "Formatted at 2025-01-15 02:53:53 on dist-test-slave-1lv2"
W20250115 02:53:53.074024 12798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:44903: Network error: Client connection negotiation failed: client connection to 127.2.36.124:44903: connect: Connection refused (error 111)
I20250115 02:53:53.078719 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250115 02:53:53.082077 12863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:53.084049 12875 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:53.085039 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250115 02:53:53.085436 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "e800fe9800dc4ac8808518c32ff6c7aa"
format_stamp: "Formatted at 2025-01-15 02:53:53 on dist-test-slave-1lv2"
I20250115 02:53:53.085781 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250115 02:53:53.087389 12863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:44903: Network error: Client connection negotiation failed: client connection to 127.2.36.124:44903: connect: Connection refused (error 111)
I20250115 02:53:53.094532 12798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } attempt: 1
W20250115 02:53:53.098549 12798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:44903: Network error: Client connection negotiation failed: client connection to 127.2.36.124:44903: connect: Connection refused (error 111)
I20250115 02:53:53.130954 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:53.132228 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:53.132838 12863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } attempt: 1
W20250115 02:53:53.137140 12863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:44903: Network error: Client connection negotiation failed: client connection to 127.2.36.124:44903: connect: Connection refused (error 111)
I20250115 02:53:53.148121 12798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } attempt: 2
W20250115 02:53:53.152792 12798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:44903: Network error: Client connection negotiation failed: client connection to 127.2.36.124:44903: connect: Connection refused (error 111)
I20250115 02:53:53.171767 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:44903
I20250115 02:53:53.171883 12927 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:44903 every 8 connection(s)
I20250115 02:53:53.174834 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:53:53.175577 12928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:53.181787 12928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:53.191709 12928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:53.200673 12928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:53.211623 12863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } attempt: 2
I20250115 02:53:53.216786 12928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa: Bootstrap starting.
I20250115 02:53:53.222281 12798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } attempt: 3
I20250115 02:53:53.222702 12928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:53.227547 12928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa: No bootstrap required, opened a new log
I20250115 02:53:53.228503 12863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c: Bootstrap starting.
I20250115 02:53:53.230057 12928 raft_consensus.cc:357] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } }
I20250115 02:53:53.230811 12928 raft_consensus.cc:383] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:53.231132 12928 raft_consensus.cc:738] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e800fe9800dc4ac8808518c32ff6c7aa, State: Initialized, Role: FOLLOWER
I20250115 02:53:53.231848 12928 consensus_queue.cc:260] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } }
I20250115 02:53:53.234580 12863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:53.235728 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } } }
I20250115 02:53:53.236567 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:53.237399 12928 sys_catalog.cc:564] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:53.239894 12863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c: No bootstrap required, opened a new log
I20250115 02:53:53.241276 12798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6: Bootstrap starting.
I20250115 02:53:53.242663 12863 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } }
I20250115 02:53:53.243487 12863 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:53.243780 12863 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc82e9e96d04610b32c8a12c6973b0c, State: Initialized, Role: FOLLOWER
I20250115 02:53:53.244503 12863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } }
I20250115 02:53:53.248410 12798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:53.250797 12941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } } }
I20250115 02:53:53.251678 12941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:53.252575 12863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [sys.catalog]: configured and running, proceeding with master startup.
W20250115 02:53:53.253990 12946 catalog_manager.cc:1559] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:53.254281 12946 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:53.254633 12798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6: No bootstrap required, opened a new log
I20250115 02:53:53.257776 12798 raft_consensus.cc:357] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } }
I20250115 02:53:53.258734 12798 raft_consensus.cc:383] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:53.259105 12798 raft_consensus.cc:738] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552ffd497bfd4e3b9bda39b14d1610a6, State: Initialized, Role: FOLLOWER
I20250115 02:53:53.259822 12798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } }
I20250115 02:53:53.262377 12954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } } }
I20250115 02:53:53.263262 12954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:53.264201 12798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [sys.catalog]: configured and running, proceeding with master startup.
W20250115 02:53:53.268517 12959 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:53.268828 12959 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:53.275266 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:53:53.275529 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:53:53.276917 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:53.278213 12970 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:53.278616 12970 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:53:53.283334 12971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:53.284015 12972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:53.286357 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:53.286599 12974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:53.287505 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:53.287737 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:53.287874 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909633287864 us; error 0 us; skew 500 ppm
I20250115 02:53:53.288383 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:53.290657 2193 webserver.cc:458] Webserver started at http://127.2.36.65:37765/ using document root <none> and password file <none>
I20250115 02:53:53.291155 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:53.291337 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:53.291568 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:53.292591 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "ddd8dc487787487fb8185ac2a85a73ef"
format_stamp: "Formatted at 2025-01-15 02:53:53 on dist-test-slave-1lv2"
I20250115 02:53:53.296775 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:53:53.299983 12979 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:53.300735 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:53:53.301023 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "ddd8dc487787487fb8185ac2a85a73ef"
format_stamp: "Formatted at 2025-01-15 02:53:53 on dist-test-slave-1lv2"
I20250115 02:53:53.301321 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:53.343840 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:53.345017 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:53.346417 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:53.348740 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:53.348945 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:53.349164 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:53.349319 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:53.388010 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:34023
I20250115 02:53:53.388104 13041 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:34023 every 8 connection(s)
I20250115 02:53:53.394874 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:53.409351 13043 heartbeater.cc:346] Connected to a master server at 127.2.36.125:43219
I20250115 02:53:53.409615 13042 heartbeater.cc:346] Connected to a master server at 127.2.36.124:44903
I20250115 02:53:53.409883 13043 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:53.410092 13042 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:53.410954 13043 heartbeater.cc:509] Master 127.2.36.125:43219 requested a full tablet report, sending...
I20250115 02:53:53.411031 13042 heartbeater.cc:509] Master 127.2.36.124:44903 requested a full tablet report, sending...
W20250115 02:53:53.415570 13049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:53.415692 12893 ts_manager.cc:194] Registered new tserver with Master: ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023)
W20250115 02:53:53.416570 13050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:53.417778 12828 ts_manager.cc:194] Registered new tserver with Master: ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023)
I20250115 02:53:53.418277 13044 heartbeater.cc:346] Connected to a master server at 127.2.36.126:35607
I20250115 02:53:53.418740 13044 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:53.419708 13044 heartbeater.cc:509] Master 127.2.36.126:35607 requested a full tablet report, sending...
I20250115 02:53:53.422122 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:53.422523 13052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:53.422633 12763 ts_manager.cc:194] Registered new tserver with Master: ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023)
I20250115 02:53:53.423621 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:53.423862 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:53.424063 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909633424049 us; error 0 us; skew 500 ppm
I20250115 02:53:53.424595 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:53.426820 2193 webserver.cc:458] Webserver started at http://127.2.36.66:38869/ using document root <none> and password file <none>
I20250115 02:53:53.427243 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:53.427400 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:53.427595 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:53.428714 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "5f49b8359ab441c49daeffdbdbac10c7"
format_stamp: "Formatted at 2025-01-15 02:53:53 on dist-test-slave-1lv2"
I20250115 02:53:53.434248 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250115 02:53:53.437995 13057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:53.438710 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:53.439038 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "5f49b8359ab441c49daeffdbdbac10c7"
format_stamp: "Formatted at 2025-01-15 02:53:53 on dist-test-slave-1lv2"
I20250115 02:53:53.439314 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:53.450799 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:53.451938 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:53.453543 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:53.456485 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:53.456748 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:53.456997 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:53.457201 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:53.458101 12941 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:53.458607 12941 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } }
I20250115 02:53:53.460454 12941 leader_election.cc:290] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 552ffd497bfd4e3b9bda39b14d1610a6 (127.2.36.126:35607), e800fe9800dc4ac8808518c32ff6c7aa (127.2.36.124:44903)
I20250115 02:53:53.461530 12773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" is_pre_election: true
I20250115 02:53:53.461930 12903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" is_pre_election: true
I20250115 02:53:53.462738 12773 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dc82e9e96d04610b32c8a12c6973b0c in term 0.
I20250115 02:53:53.462661 12903 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dc82e9e96d04610b32c8a12c6973b0c in term 0.
I20250115 02:53:53.464320 12815 leader_election.cc:304] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 552ffd497bfd4e3b9bda39b14d1610a6, 8dc82e9e96d04610b32c8a12c6973b0c; no voters:
I20250115 02:53:53.465700 12941 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:53.466058 12941 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:53.466411 12941 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:53.476532 12941 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } }
I20250115 02:53:53.480221 12941 leader_election.cc:290] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [CANDIDATE]: Term 1 election: Requested vote from peers 552ffd497bfd4e3b9bda39b14d1610a6 (127.2.36.126:35607), e800fe9800dc4ac8808518c32ff6c7aa (127.2.36.124:44903)
I20250115 02:53:53.481509 12773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "552ffd497bfd4e3b9bda39b14d1610a6"
I20250115 02:53:53.482097 12773 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:53.483130 12903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e800fe9800dc4ac8808518c32ff6c7aa"
I20250115 02:53:53.483846 12903 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:53.488107 12773 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dc82e9e96d04610b32c8a12c6973b0c in term 1.
I20250115 02:53:53.489353 12815 leader_election.cc:304] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 552ffd497bfd4e3b9bda39b14d1610a6, 8dc82e9e96d04610b32c8a12c6973b0c; no voters:
I20250115 02:53:53.491039 12941 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:53.492434 12903 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dc82e9e96d04610b32c8a12c6973b0c in term 1.
I20250115 02:53:53.494551 12941 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 1 LEADER]: Becoming Leader. State: Replica: 8dc82e9e96d04610b32c8a12c6973b0c, State: Running, Role: LEADER
I20250115 02:53:53.495371 12941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } }
I20250115 02:53:53.501983 13080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dc82e9e96d04610b32c8a12c6973b0c. Latest consensus state: current_term: 1 leader_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } } }
I20250115 02:53:53.502640 13080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:53.504549 13085 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:53.512446 13085 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:53.524742 12903 raft_consensus.cc:1270] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [term 1 FOLLOWER]: Refusing update from remote peer 8dc82e9e96d04610b32c8a12c6973b0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:53.528771 13080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:53.530575 12773 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [term 1 FOLLOWER]: Refusing update from remote peer 8dc82e9e96d04610b32c8a12c6973b0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:53.532663 13080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:53.539716 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dc82e9e96d04610b32c8a12c6973b0c. Latest consensus state: current_term: 1 leader_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } } }
I20250115 02:53:53.540577 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:53.552397 13080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } } }
I20250115 02:53:53.553423 13080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:53.554847 13085 catalog_manager.cc:1348] Generated new cluster ID: cea56b01240f458ea05899503d8b78a7
I20250115 02:53:53.555262 13085 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:53.555020 12954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dc82e9e96d04610b32c8a12c6973b0c. Latest consensus state: current_term: 1 leader_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } } }
I20250115 02:53:53.555756 12954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:53.561501 12941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } } }
I20250115 02:53:53.562345 12941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:53.563534 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } } }
I20250115 02:53:53.564365 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:53.580067 12954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552ffd497bfd4e3b9bda39b14d1610a6" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 35607 } } peers { permanent_uuid: "8dc82e9e96d04610b32c8a12c6973b0c" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43219 } } peers { permanent_uuid: "e800fe9800dc4ac8808518c32ff6c7aa" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 44903 } } }
I20250115 02:53:53.580902 12954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:53.590914 13085 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:53:53.592535 13085 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:53.618795 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:42877
I20250115 02:53:53.619026 13130 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:42877 every 8 connection(s)
I20250115 02:53:53.635723 13085 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c: Generated new TSK 0
I20250115 02:53:53.636919 13085 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:53.657469 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:53.660127 12828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55904
I20250115 02:53:53.673288 13043 heartbeater.cc:501] Master 127.2.36.125:43219 was elected leader, sending a full tablet report...
I20250115 02:53:53.677348 13131 heartbeater.cc:346] Connected to a master server at 127.2.36.124:44903
I20250115 02:53:53.677695 13131 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:53.678593 13131 heartbeater.cc:509] Master 127.2.36.124:44903 requested a full tablet report, sending...
W20250115 02:53:53.681550 13138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:53.684722 12893 ts_manager.cc:194] Registered new tserver with Master: 5f49b8359ab441c49daeffdbdbac10c7 (127.2.36.66:42877)
I20250115 02:53:53.688046 13132 heartbeater.cc:346] Connected to a master server at 127.2.36.125:43219
I20250115 02:53:53.688809 13132 heartbeater.cc:463] Registering TS with master...
W20250115 02:53:53.689425 13139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:53.689690 13132 heartbeater.cc:509] Master 127.2.36.125:43219 requested a full tablet report, sending...
I20250115 02:53:53.692883 13133 heartbeater.cc:346] Connected to a master server at 127.2.36.126:35607
I20250115 02:53:53.693291 13133 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:53.693756 12828 ts_manager.cc:194] Registered new tserver with Master: 5f49b8359ab441c49daeffdbdbac10c7 (127.2.36.66:42877)
I20250115 02:53:53.694237 13133 heartbeater.cc:509] Master 127.2.36.126:35607 requested a full tablet report, sending...
I20250115 02:53:53.695672 12828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55916
I20250115 02:53:53.697281 12763 ts_manager.cc:194] Registered new tserver with Master: 5f49b8359ab441c49daeffdbdbac10c7 (127.2.36.66:42877)
I20250115 02:53:53.699601 2193 server_base.cc:1034] running on GCE node
W20250115 02:53:53.700878 13141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:53.701874 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:53.702091 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:53.702260 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909633702246 us; error 0 us; skew 500 ppm
I20250115 02:53:53.702852 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:53.705725 2193 webserver.cc:458] Webserver started at http://127.2.36.67:33755/ using document root <none> and password file <none>
I20250115 02:53:53.706333 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:53.706533 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:53.706799 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:53.707811 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "394e289191004b0f8cca0584e736d09e"
format_stamp: "Formatted at 2025-01-15 02:53:53 on dist-test-slave-1lv2"
I20250115 02:53:53.708745 13132 heartbeater.cc:501] Master 127.2.36.125:43219 was elected leader, sending a full tablet report...
I20250115 02:53:53.713477 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250115 02:53:53.717355 13146 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:53.718288 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250115 02:53:53.718700 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "394e289191004b0f8cca0584e736d09e"
format_stamp: "Formatted at 2025-01-15 02:53:53 on dist-test-slave-1lv2"
I20250115 02:53:53.719096 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:53.742610 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:53.743944 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:53.745893 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:53.748090 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:53.748348 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:53.748576 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:53.748728 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:53.811131 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:44373
I20250115 02:53:53.811213 13208 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:44373 every 8 connection(s)
I20250115 02:53:53.833554 13209 heartbeater.cc:346] Connected to a master server at 127.2.36.124:44903
I20250115 02:53:53.833976 13209 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:53.834856 13209 heartbeater.cc:509] Master 127.2.36.124:44903 requested a full tablet report, sending...
I20250115 02:53:53.840706 12893 ts_manager.cc:194] Registered new tserver with Master: 394e289191004b0f8cca0584e736d09e (127.2.36.67:44373)
I20250115 02:53:53.843467 13211 heartbeater.cc:346] Connected to a master server at 127.2.36.126:35607
I20250115 02:53:53.843779 13211 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:53.844548 13211 heartbeater.cc:509] Master 127.2.36.126:35607 requested a full tablet report, sending...
I20250115 02:53:53.845474 13210 heartbeater.cc:346] Connected to a master server at 127.2.36.125:43219
I20250115 02:53:53.845885 13210 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:53.846660 13210 heartbeater.cc:509] Master 127.2.36.125:43219 requested a full tablet report, sending...
I20250115 02:53:53.847257 12762 ts_manager.cc:194] Registered new tserver with Master: 394e289191004b0f8cca0584e736d09e (127.2.36.67:44373)
I20250115 02:53:53.849066 12827 ts_manager.cc:194] Registered new tserver with Master: 394e289191004b0f8cca0584e736d09e (127.2.36.67:44373)
I20250115 02:53:53.851064 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.031674883s
I20250115 02:53:53.851114 12827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55924
I20250115 02:53:53.864629 13210 heartbeater.cc:501] Master 127.2.36.125:43219 was elected leader, sending a full tablet report...
I20250115 02:53:53.899061 12828 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55928:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:53:53.902346 12828 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:53:53.999490 13173 tablet_service.cc:1467] Processing CreateTablet for tablet b0fd51eff7cc464aa4463b42a31b4bbb (DEFAULT_TABLE table=ksck-test-table [id=b6083a12e66543938f1e5bf8e07a14b9]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:54.000957 13173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0fd51eff7cc464aa4463b42a31b4bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:54.003726 13174 tablet_service.cc:1467] Processing CreateTablet for tablet db65c67328254bac96715cf661f75ebf (DEFAULT_TABLE table=ksck-test-table [id=b6083a12e66543938f1e5bf8e07a14b9]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:54.005074 13174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db65c67328254bac96715cf661f75ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:54.012399 13172 tablet_service.cc:1467] Processing CreateTablet for tablet c261d1abd4ad403981479f142f9229ff (DEFAULT_TABLE table=ksck-test-table [id=b6083a12e66543938f1e5bf8e07a14b9]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:54.013352 13088 tablet_service.cc:1467] Processing CreateTablet for tablet db65c67328254bac96715cf661f75ebf (DEFAULT_TABLE table=ksck-test-table [id=b6083a12e66543938f1e5bf8e07a14b9]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:54.013536 13087 tablet_service.cc:1467] Processing CreateTablet for tablet b0fd51eff7cc464aa4463b42a31b4bbb (DEFAULT_TABLE table=ksck-test-table [id=b6083a12e66543938f1e5bf8e07a14b9]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:54.013855 13172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c261d1abd4ad403981479f142f9229ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:54.014901 13087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0fd51eff7cc464aa4463b42a31b4bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:54.017952 13088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db65c67328254bac96715cf661f75ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:54.021425 13086 tablet_service.cc:1467] Processing CreateTablet for tablet c261d1abd4ad403981479f142f9229ff (DEFAULT_TABLE table=ksck-test-table [id=b6083a12e66543938f1e5bf8e07a14b9]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:54.022799 13086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c261d1abd4ad403981479f142f9229ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:54.051842 13229 tablet_bootstrap.cc:492] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7: Bootstrap starting.
I20250115 02:53:54.064025 13230 tablet_bootstrap.cc:492] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e: Bootstrap starting.
I20250115 02:53:54.071563 13006 tablet_service.cc:1467] Processing CreateTablet for tablet b0fd51eff7cc464aa4463b42a31b4bbb (DEFAULT_TABLE table=ksck-test-table [id=b6083a12e66543938f1e5bf8e07a14b9]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:54.072955 13006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0fd51eff7cc464aa4463b42a31b4bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:54.075637 13005 tablet_service.cc:1467] Processing CreateTablet for tablet c261d1abd4ad403981479f142f9229ff (DEFAULT_TABLE table=ksck-test-table [id=b6083a12e66543938f1e5bf8e07a14b9]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:54.076704 13005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c261d1abd4ad403981479f142f9229ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:54.078765 13229 tablet_bootstrap.cc:654] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:54.079378 13007 tablet_service.cc:1467] Processing CreateTablet for tablet db65c67328254bac96715cf661f75ebf (DEFAULT_TABLE table=ksck-test-table [id=b6083a12e66543938f1e5bf8e07a14b9]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:54.080410 13007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db65c67328254bac96715cf661f75ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:54.081737 13230 tablet_bootstrap.cc:654] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:54.092473 13229 tablet_bootstrap.cc:492] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7: No bootstrap required, opened a new log
I20250115 02:53:54.093039 13229 ts_tablet_manager.cc:1397] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.011s
I20250115 02:53:54.095978 13229 raft_consensus.cc:357] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.096961 13229 raft_consensus.cc:383] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:54.097333 13229 raft_consensus.cc:738] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49b8359ab441c49daeffdbdbac10c7, State: Initialized, Role: FOLLOWER
I20250115 02:53:54.098368 13229 consensus_queue.cc:260] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.136440 13230 tablet_bootstrap.cc:492] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e: No bootstrap required, opened a new log
I20250115 02:53:54.137064 13230 ts_tablet_manager.cc:1397] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.010s
I20250115 02:53:54.170228 13234 tablet_bootstrap.cc:492] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef: Bootstrap starting.
I20250115 02:53:54.181190 13229 ts_tablet_manager.cc:1428] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7: Time spent starting tablet: real 0.088s user 0.013s sys 0.012s
I20250115 02:53:54.182489 13229 tablet_bootstrap.cc:492] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7: Bootstrap starting.
I20250115 02:53:54.188133 13234 tablet_bootstrap.cc:654] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:54.191557 13230 raft_consensus.cc:357] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } }
I20250115 02:53:54.193147 13230 raft_consensus.cc:383] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:54.193542 13230 raft_consensus.cc:738] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394e289191004b0f8cca0584e736d09e, State: Initialized, Role: FOLLOWER
I20250115 02:53:54.194417 13230 consensus_queue.cc:260] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } }
I20250115 02:53:54.197736 13229 tablet_bootstrap.cc:654] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:54.201617 13234 tablet_bootstrap.cc:492] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef: No bootstrap required, opened a new log
I20250115 02:53:54.207240 13234 ts_tablet_manager.cc:1397] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.009s
I20250115 02:53:54.210721 13234 raft_consensus.cc:357] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.212028 13234 raft_consensus.cc:383] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:54.212366 13229 tablet_bootstrap.cc:492] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7: No bootstrap required, opened a new log
I20250115 02:53:54.213034 13229 ts_tablet_manager.cc:1397] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.001s
I20250115 02:53:54.212395 13234 raft_consensus.cc:738] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd8dc487787487fb8185ac2a85a73ef, State: Initialized, Role: FOLLOWER
I20250115 02:53:54.214483 13234 consensus_queue.cc:260] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.222707 13230 ts_tablet_manager.cc:1428] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e: Time spent starting tablet: real 0.085s user 0.002s sys 0.013s
I20250115 02:53:54.223628 13229 raft_consensus.cc:357] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } }
I20250115 02:53:54.224514 13229 raft_consensus.cc:383] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:54.224812 13229 raft_consensus.cc:738] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49b8359ab441c49daeffdbdbac10c7, State: Initialized, Role: FOLLOWER
I20250115 02:53:54.225534 13229 consensus_queue.cc:260] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } }
I20250115 02:53:54.228086 13229 ts_tablet_manager.cc:1428] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250115 02:53:54.229220 13229 tablet_bootstrap.cc:492] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7: Bootstrap starting.
I20250115 02:53:54.236912 13229 tablet_bootstrap.cc:654] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:54.237455 13234 ts_tablet_manager.cc:1428] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef: Time spent starting tablet: real 0.029s user 0.009s sys 0.009s
I20250115 02:53:54.238783 13234 tablet_bootstrap.cc:492] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef: Bootstrap starting.
I20250115 02:53:54.248749 13229 tablet_bootstrap.cc:492] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7: No bootstrap required, opened a new log
I20250115 02:53:54.249403 13229 ts_tablet_manager.cc:1397] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20250115 02:53:54.252271 13229 raft_consensus.cc:357] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.253299 13229 raft_consensus.cc:383] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:54.253648 13229 raft_consensus.cc:738] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49b8359ab441c49daeffdbdbac10c7, State: Initialized, Role: FOLLOWER
I20250115 02:53:54.256538 13234 tablet_bootstrap.cc:654] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:54.259923 12946 catalog_manager.cc:1260] Loaded cluster ID: cea56b01240f458ea05899503d8b78a7
I20250115 02:53:54.260334 12946 catalog_manager.cc:1553] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa: loading cluster ID for follower catalog manager: success
I20250115 02:53:54.262187 13230 tablet_bootstrap.cc:492] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e: Bootstrap starting.
I20250115 02:53:54.267271 12946 catalog_manager.cc:1575] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa: acquiring CA information for follower catalog manager: success
I20250115 02:53:54.254490 13229 consensus_queue.cc:260] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.274194 13229 ts_tablet_manager.cc:1428] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20250115 02:53:54.274670 13233 raft_consensus.cc:491] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:54.275166 13233 raft_consensus.cc:513] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.274443 13236 raft_consensus.cc:491] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:54.279673 13236 raft_consensus.cc:513] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } }
I20250115 02:53:54.280840 13233 leader_election.cc:290] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023), 394e289191004b0f8cca0584e736d09e (127.2.36.67:44373)
I20250115 02:53:54.299420 12970 catalog_manager.cc:1260] Loaded cluster ID: cea56b01240f458ea05899503d8b78a7
I20250115 02:53:54.307754 12970 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6: loading cluster ID for follower catalog manager: success
I20250115 02:53:54.317765 13236 leader_election.cc:290] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023), 5f49b8359ab441c49daeffdbdbac10c7 (127.2.36.66:42877)
I20250115 02:53:54.320767 13230 tablet_bootstrap.cc:654] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:54.329691 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65c67328254bac96715cf661f75ebf" candidate_uuid: "394e289191004b0f8cca0584e736d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd8dc487787487fb8185ac2a85a73ef" is_pre_election: true
I20250115 02:53:54.330140 13099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65c67328254bac96715cf661f75ebf" candidate_uuid: "394e289191004b0f8cca0584e736d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49b8359ab441c49daeffdbdbac10c7" is_pre_election: true
I20250115 02:53:54.330886 13099 raft_consensus.cc:2463] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 394e289191004b0f8cca0584e736d09e in term 0.
W20250115 02:53:54.332248 13150 leader_election.cc:343] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:54.338536 13233 raft_consensus.cc:491] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:54.337173 13147 leader_election.cc:304] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 394e289191004b0f8cca0584e736d09e, 5f49b8359ab441c49daeffdbdbac10c7; no voters: ddd8dc487787487fb8185ac2a85a73ef
I20250115 02:53:54.339083 13233 raft_consensus.cc:513] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.340050 13236 raft_consensus.cc:2798] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:54.340447 13236 raft_consensus.cc:491] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:54.340994 13236 raft_consensus.cc:3054] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:54.341343 13233 leader_election.cc:290] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023), 394e289191004b0f8cca0584e736d09e (127.2.36.67:44373)
I20250115 02:53:54.340623 12946 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:54.344666 12970 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6: acquiring CA information for follower catalog manager: success
I20250115 02:53:54.345970 13234 tablet_bootstrap.cc:492] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef: No bootstrap required, opened a new log
I20250115 02:53:54.346568 13234 ts_tablet_manager.cc:1397] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef: Time spent bootstrapping tablet: real 0.108s user 0.019s sys 0.020s
I20250115 02:53:54.349548 13234 raft_consensus.cc:357] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } }
I20250115 02:53:54.350095 12970 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:54.349874 13236 raft_consensus.cc:513] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } }
I20250115 02:53:54.360361 13238 raft_consensus.cc:491] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:54.365309 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c261d1abd4ad403981479f142f9229ff" candidate_uuid: "5f49b8359ab441c49daeffdbdbac10c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd8dc487787487fb8185ac2a85a73ef" is_pre_election: true
I20250115 02:53:54.370280 13230 tablet_bootstrap.cc:492] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e: No bootstrap required, opened a new log
I20250115 02:53:54.365752 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0fd51eff7cc464aa4463b42a31b4bbb" candidate_uuid: "5f49b8359ab441c49daeffdbdbac10c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd8dc487787487fb8185ac2a85a73ef" is_pre_election: true
I20250115 02:53:54.371047 13234 raft_consensus.cc:383] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:54.373085 13236 leader_election.cc:290] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [CANDIDATE]: Term 1 election: Requested vote from peers ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023), 5f49b8359ab441c49daeffdbdbac10c7 (127.2.36.66:42877)
I20250115 02:53:54.372530 13230 ts_tablet_manager.cc:1397] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e: Time spent bootstrapping tablet: real 0.111s user 0.013s sys 0.000s
I20250115 02:53:54.375329 13234 raft_consensus.cc:738] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd8dc487787487fb8185ac2a85a73ef, State: Initialized, Role: FOLLOWER
I20250115 02:53:54.375875 13099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65c67328254bac96715cf661f75ebf" candidate_uuid: "394e289191004b0f8cca0584e736d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49b8359ab441c49daeffdbdbac10c7"
I20250115 02:53:54.376580 13234 consensus_queue.cc:260] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } }
I20250115 02:53:54.377274 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65c67328254bac96715cf661f75ebf" candidate_uuid: "394e289191004b0f8cca0584e736d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd8dc487787487fb8185ac2a85a73ef"
I20250115 02:53:54.375211 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c261d1abd4ad403981479f142f9229ff" candidate_uuid: "5f49b8359ab441c49daeffdbdbac10c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "394e289191004b0f8cca0584e736d09e" is_pre_election: true
I20250115 02:53:54.376498 13099 raft_consensus.cc:3054] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:54.376561 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0fd51eff7cc464aa4463b42a31b4bbb" candidate_uuid: "5f49b8359ab441c49daeffdbdbac10c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "394e289191004b0f8cca0584e736d09e" is_pre_election: true
I20250115 02:53:54.371908 13238 raft_consensus.cc:513] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.385170 13230 raft_consensus.cc:357] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.386026 13230 raft_consensus.cc:383] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:54.386350 13230 raft_consensus.cc:738] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394e289191004b0f8cca0584e736d09e, State: Initialized, Role: FOLLOWER
W20250115 02:53:54.386945 13061 leader_election.cc:343] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:54.387097 13230 consensus_queue.cc:260] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.389645 13230 ts_tablet_manager.cc:1428] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e: Time spent starting tablet: real 0.014s user 0.007s sys 0.004s
I20250115 02:53:54.390131 13238 leader_election.cc:290] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f49b8359ab441c49daeffdbdbac10c7 (127.2.36.66:42877), 394e289191004b0f8cca0584e736d09e (127.2.36.67:44373)
I20250115 02:53:54.390611 13017 raft_consensus.cc:2463] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49b8359ab441c49daeffdbdbac10c7 in term 0.
I20250115 02:53:54.392596 13183 raft_consensus.cc:2463] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49b8359ab441c49daeffdbdbac10c7 in term 0.
I20250115 02:53:54.393539 13099 raft_consensus.cc:2463] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 394e289191004b0f8cca0584e736d09e in term 1.
I20250115 02:53:54.394733 13147 leader_election.cc:304] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 394e289191004b0f8cca0584e736d09e, 5f49b8359ab441c49daeffdbdbac10c7; no voters:
I20250115 02:53:54.395714 13236 raft_consensus.cc:2798] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 1 FOLLOWER]: Leader election won for term 1
W20250115 02:53:54.392128 13059 leader_election.cc:343] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 394e289191004b0f8cca0584e736d09e (127.2.36.67:44373): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:54.398717 13059 leader_election.cc:304] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 394e289191004b0f8cca0584e736d09e, 5f49b8359ab441c49daeffdbdbac10c7; no voters: ddd8dc487787487fb8185ac2a85a73ef
I20250115 02:53:54.399459 13230 tablet_bootstrap.cc:492] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e: Bootstrap starting.
I20250115 02:53:54.401474 13233 raft_consensus.cc:2798] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:54.401865 13233 raft_consensus.cc:491] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:54.402172 13233 raft_consensus.cc:3054] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:54.403043 13061 leader_election.cc:304] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f49b8359ab441c49daeffdbdbac10c7, ddd8dc487787487fb8185ac2a85a73ef; no voters: 394e289191004b0f8cca0584e736d09e
I20250115 02:53:54.406203 13016 raft_consensus.cc:3054] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:54.409344 13248 raft_consensus.cc:2798] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:54.409943 13248 raft_consensus.cc:491] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:54.410307 13248 raft_consensus.cc:3054] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:54.413093 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c261d1abd4ad403981479f142f9229ff" candidate_uuid: "ddd8dc487787487fb8185ac2a85a73ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "394e289191004b0f8cca0584e736d09e" is_pre_election: true
W20250115 02:53:54.414671 12981 leader_election.cc:343] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 394e289191004b0f8cca0584e736d09e (127.2.36.67:44373): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:54.416486 13016 raft_consensus.cc:2463] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 394e289191004b0f8cca0584e736d09e in term 1.
I20250115 02:53:54.420320 13234 ts_tablet_manager.cc:1428] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef: Time spent starting tablet: real 0.073s user 0.000s sys 0.006s
I20250115 02:53:54.427800 13234 tablet_bootstrap.cc:492] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef: Bootstrap starting.
I20250115 02:53:54.432049 13233 raft_consensus.cc:513] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.426028 13248 raft_consensus.cc:513] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.435104 13248 leader_election.cc:290] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [CANDIDATE]: Term 1 election: Requested vote from peers ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023), 394e289191004b0f8cca0584e736d09e (127.2.36.67:44373)
I20250115 02:53:54.436075 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0fd51eff7cc464aa4463b42a31b4bbb" candidate_uuid: "5f49b8359ab441c49daeffdbdbac10c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "394e289191004b0f8cca0584e736d09e"
I20250115 02:53:54.436465 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c261d1abd4ad403981479f142f9229ff" candidate_uuid: "5f49b8359ab441c49daeffdbdbac10c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "394e289191004b0f8cca0584e736d09e"
I20250115 02:53:54.436800 13183 raft_consensus.cc:3054] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Advancing to term 1
W20250115 02:53:54.437870 13059 leader_election.cc:343] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 394e289191004b0f8cca0584e736d09e (127.2.36.67:44373): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:54.438423 13233 leader_election.cc:290] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [CANDIDATE]: Term 1 election: Requested vote from peers ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023), 394e289191004b0f8cca0584e736d09e (127.2.36.67:44373)
I20250115 02:53:54.443260 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0fd51eff7cc464aa4463b42a31b4bbb" candidate_uuid: "5f49b8359ab441c49daeffdbdbac10c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd8dc487787487fb8185ac2a85a73ef"
I20250115 02:53:54.443584 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c261d1abd4ad403981479f142f9229ff" candidate_uuid: "5f49b8359ab441c49daeffdbdbac10c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd8dc487787487fb8185ac2a85a73ef"
I20250115 02:53:54.444175 13017 raft_consensus.cc:3054] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:54.446312 13183 raft_consensus.cc:2463] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49b8359ab441c49daeffdbdbac10c7 in term 1.
I20250115 02:53:54.447230 13059 leader_election.cc:304] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 394e289191004b0f8cca0584e736d09e, 5f49b8359ab441c49daeffdbdbac10c7; no voters:
I20250115 02:53:54.447877 13233 raft_consensus.cc:2798] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 FOLLOWER]: Leader election won for term 1
W20250115 02:53:54.450922 13061 leader_election.cc:343] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:54.449275 13230 tablet_bootstrap.cc:654] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:54.452546 13233 raft_consensus.cc:695] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 LEADER]: Becoming Leader. State: Replica: 5f49b8359ab441c49daeffdbdbac10c7, State: Running, Role: LEADER
I20250115 02:53:54.453548 13236 raft_consensus.cc:695] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 1 LEADER]: Becoming Leader. State: Replica: 394e289191004b0f8cca0584e736d09e, State: Running, Role: LEADER
I20250115 02:53:54.475071 13099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c261d1abd4ad403981479f142f9229ff" candidate_uuid: "ddd8dc487787487fb8185ac2a85a73ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49b8359ab441c49daeffdbdbac10c7" is_pre_election: true
I20250115 02:53:54.475903 13099 raft_consensus.cc:2388] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ddd8dc487787487fb8185ac2a85a73ef in current term 1: Already voted for candidate 5f49b8359ab441c49daeffdbdbac10c7 in this term.
I20250115 02:53:54.478551 12980 leader_election.cc:304] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddd8dc487787487fb8185ac2a85a73ef; no voters: 394e289191004b0f8cca0584e736d09e, 5f49b8359ab441c49daeffdbdbac10c7
I20250115 02:53:54.471860 13233 consensus_queue.cc:237] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.473820 13236 consensus_queue.cc:237] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } }
I20250115 02:53:54.488904 13230 tablet_bootstrap.cc:492] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e: No bootstrap required, opened a new log
I20250115 02:53:54.489334 13230 ts_tablet_manager.cc:1397] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e: Time spent bootstrapping tablet: real 0.090s user 0.010s sys 0.006s
I20250115 02:53:54.491715 13017 raft_consensus.cc:2463] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49b8359ab441c49daeffdbdbac10c7 in term 1.
I20250115 02:53:54.492565 13238 raft_consensus.cc:2743] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:53:54.495093 13061 leader_election.cc:304] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f49b8359ab441c49daeffdbdbac10c7, ddd8dc487787487fb8185ac2a85a73ef; no voters: 394e289191004b0f8cca0584e736d09e
I20250115 02:53:54.491789 13230 raft_consensus.cc:357] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.498726 13230 raft_consensus.cc:383] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:54.499156 13230 raft_consensus.cc:738] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394e289191004b0f8cca0584e736d09e, State: Initialized, Role: FOLLOWER
I20250115 02:53:54.500008 13230 consensus_queue.cc:260] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.496627 13233 raft_consensus.cc:2798] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:54.502898 13233 raft_consensus.cc:695] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 LEADER]: Becoming Leader. State: Replica: 5f49b8359ab441c49daeffdbdbac10c7, State: Running, Role: LEADER
I20250115 02:53:54.503023 13230 ts_tablet_manager.cc:1428] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e: Time spent starting tablet: real 0.013s user 0.004s sys 0.005s
I20250115 02:53:54.503769 13233 consensus_queue.cc:237] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.504583 13234 tablet_bootstrap.cc:654] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:54.511092 12828 catalog_manager.cc:5526] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e reported cstate change: term changed from 0 to 1, leader changed from <none> to 394e289191004b0f8cca0584e736d09e (127.2.36.67). New cstate: current_term: 1 leader_uuid: "394e289191004b0f8cca0584e736d09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:54.519589 12826 catalog_manager.cc:5526] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f49b8359ab441c49daeffdbdbac10c7 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "5f49b8359ab441c49daeffdbdbac10c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:54.542663 13234 tablet_bootstrap.cc:492] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef: No bootstrap required, opened a new log
I20250115 02:53:54.543311 13234 ts_tablet_manager.cc:1397] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef: Time spent bootstrapping tablet: real 0.116s user 0.008s sys 0.006s
I20250115 02:53:54.546286 13234 raft_consensus.cc:357] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.547080 13234 raft_consensus.cc:383] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:54.547317 13234 raft_consensus.cc:738] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd8dc487787487fb8185ac2a85a73ef, State: Initialized, Role: FOLLOWER
I20250115 02:53:54.547900 13234 consensus_queue.cc:260] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.550129 13234 ts_tablet_manager.cc:1428] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250115 02:53:54.577188 13238 raft_consensus.cc:491] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:54.577845 13238 raft_consensus.cc:513] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.580356 13238 leader_election.cc:290] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f49b8359ab441c49daeffdbdbac10c7 (127.2.36.66:42877), 394e289191004b0f8cca0584e736d09e (127.2.36.67:44373)
I20250115 02:53:54.584111 13099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0fd51eff7cc464aa4463b42a31b4bbb" candidate_uuid: "ddd8dc487787487fb8185ac2a85a73ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49b8359ab441c49daeffdbdbac10c7" is_pre_election: true
I20250115 02:53:54.585033 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0fd51eff7cc464aa4463b42a31b4bbb" candidate_uuid: "ddd8dc487787487fb8185ac2a85a73ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "394e289191004b0f8cca0584e736d09e" is_pre_election: true
I20250115 02:53:54.595402 13183 raft_consensus.cc:2388] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ddd8dc487787487fb8185ac2a85a73ef in current term 1: Already voted for candidate 5f49b8359ab441c49daeffdbdbac10c7 in this term.
I20250115 02:53:54.596909 12981 leader_election.cc:304] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddd8dc487787487fb8185ac2a85a73ef; no voters: 394e289191004b0f8cca0584e736d09e, 5f49b8359ab441c49daeffdbdbac10c7
I20250115 02:53:54.598487 13238 raft_consensus.cc:3054] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:54.613802 13238 raft_consensus.cc:2743] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:53:54.623211 12827 catalog_manager.cc:5526] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f49b8359ab441c49daeffdbdbac10c7 (127.2.36.66). New cstate: current_term: 1 leader_uuid: "5f49b8359ab441c49daeffdbdbac10c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:54.826033 13236 raft_consensus.cc:491] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:54.826622 13236 raft_consensus.cc:513] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 } } peers { permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 } } peers { permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 } }
I20250115 02:53:54.828811 13236 leader_election.cc:290] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023), 5f49b8359ab441c49daeffdbdbac10c7 (127.2.36.66:42877)
I20250115 02:53:54.829483 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c261d1abd4ad403981479f142f9229ff" candidate_uuid: "394e289191004b0f8cca0584e736d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd8dc487787487fb8185ac2a85a73ef" is_pre_election: true
I20250115 02:53:54.830328 13017 raft_consensus.cc:2388] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 394e289191004b0f8cca0584e736d09e in current term 1: Already voted for candidate 5f49b8359ab441c49daeffdbdbac10c7 in this term.
I20250115 02:53:54.830340 13099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c261d1abd4ad403981479f142f9229ff" candidate_uuid: "394e289191004b0f8cca0584e736d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49b8359ab441c49daeffdbdbac10c7" is_pre_election: true
I20250115 02:53:54.833598 13150 leader_election.cc:304] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 394e289191004b0f8cca0584e736d09e; no voters: 5f49b8359ab441c49daeffdbdbac10c7, ddd8dc487787487fb8185ac2a85a73ef
I20250115 02:53:54.834510 13236 raft_consensus.cc:3054] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:54.847987 13236 raft_consensus.cc:2743] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:53:54.882717 13236 consensus_queue.cc:1035] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:54.891134 13233 consensus_queue.cc:1035] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:54.933419 13248 consensus_queue.cc:1035] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:54.965247 13233 consensus_queue.cc:1035] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd8dc487787487fb8185ac2a85a73ef" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:54.994781 13233 consensus_queue.cc:1035] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "394e289191004b0f8cca0584e736d09e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 02:53:55.048833 13236 consensus_queue.cc:1035] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f49b8359ab441c49daeffdbdbac10c7" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:55.441907 13172 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:53:55.447981 13086 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250115 02:53:55.479751 13006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:53:55.847868 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:53:55.899282 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:55.900036 2193 tablet_replica.cc:331] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef: stopping tablet replica
I20250115 02:53:55.900962 2193 raft_consensus.cc:2238] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:55.901465 2193 raft_consensus.cc:2267] T b0fd51eff7cc464aa4463b42a31b4bbb P ddd8dc487787487fb8185ac2a85a73ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:55.903868 2193 tablet_replica.cc:331] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef: stopping tablet replica
I20250115 02:53:55.904376 2193 raft_consensus.cc:2238] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef [term 1 FOLLOWER]: Raft consensus shutting down.
W20250115 02:53:55.904628 13061 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 13 similar messages]
I20250115 02:53:55.904752 2193 raft_consensus.cc:2267] T db65c67328254bac96715cf661f75ebf P ddd8dc487787487fb8185ac2a85a73ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:55.907433 2193 tablet_replica.cc:331] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef: stopping tablet replica
I20250115 02:53:55.907899 2193 raft_consensus.cc:2238] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [term 1 FOLLOWER]: Raft consensus shutting down.
W20250115 02:53:55.908264 13061 consensus_peers.cc:487] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 -> Peer ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023): Couldn't send request to peer ddd8dc487787487fb8185ac2a85a73ef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250115 02:53:55.908280 2193 raft_consensus.cc:2267] T c261d1abd4ad403981479f142f9229ff P ddd8dc487787487fb8185ac2a85a73ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:55.935420 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:53:55.950888 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:53:55.980270 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:55.981009 2193 tablet_replica.cc:331] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7: stopping tablet replica
I20250115 02:53:55.981671 2193 raft_consensus.cc:2238] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 LEADER]: Raft consensus shutting down.
W20250115 02:53:55.981933 13147 consensus_peers.cc:487] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e -> Peer 5f49b8359ab441c49daeffdbdbac10c7 (127.2.36.66:42877): Couldn't send request to peer 5f49b8359ab441c49daeffdbdbac10c7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250115 02:53:55.982427 2193 raft_consensus.cc:2267] T c261d1abd4ad403981479f142f9229ff P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:55.984900 2193 tablet_replica.cc:331] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7: stopping tablet replica
I20250115 02:53:55.985569 2193 raft_consensus.cc:2238] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:55.985962 2193 raft_consensus.cc:2267] T db65c67328254bac96715cf661f75ebf P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:55.988533 2193 tablet_replica.cc:331] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7: stopping tablet replica
I20250115 02:53:55.989109 2193 raft_consensus.cc:2238] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:55.989987 2193 raft_consensus.cc:2267] T b0fd51eff7cc464aa4463b42a31b4bbb P 5f49b8359ab441c49daeffdbdbac10c7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 02:53:56.004812 13150 consensus_peers.cc:487] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e -> Peer ddd8dc487787487fb8185ac2a85a73ef (127.2.36.65:34023): Couldn't send request to peer ddd8dc487787487fb8185ac2a85a73ef. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:34023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:53:56.011668 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:53:56.028167 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:53:56.052850 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:56.053413 2193 tablet_replica.cc:331] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e: stopping tablet replica
I20250115 02:53:56.053995 2193 raft_consensus.cc:2238] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:56.054417 2193 raft_consensus.cc:2267] T c261d1abd4ad403981479f142f9229ff P 394e289191004b0f8cca0584e736d09e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:56.056546 2193 tablet_replica.cc:331] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e: stopping tablet replica
I20250115 02:53:56.057053 2193 raft_consensus.cc:2238] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:56.057438 2193 raft_consensus.cc:2267] T b0fd51eff7cc464aa4463b42a31b4bbb P 394e289191004b0f8cca0584e736d09e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:56.059213 2193 tablet_replica.cc:331] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e: stopping tablet replica
I20250115 02:53:56.059684 2193 raft_consensus.cc:2238] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:56.060515 2193 raft_consensus.cc:2267] T db65c67328254bac96715cf661f75ebf P 394e289191004b0f8cca0584e736d09e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:56.082907 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:53:56.094815 2193 master.cc:537] Master@127.2.36.126:35607 shutting down...
I20250115 02:53:56.110873 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:56.111379 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:56.111673 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 552ffd497bfd4e3b9bda39b14d1610a6: stopping tablet replica
I20250115 02:53:56.130203 2193 master.cc:559] Master@127.2.36.126:35607 shutdown complete.
I20250115 02:53:56.144595 2193 master.cc:537] Master@127.2.36.125:43219 shutting down...
I20250115 02:53:56.160610 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:56.161455 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:56.161893 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 8dc82e9e96d04610b32c8a12c6973b0c: stopping tablet replica
I20250115 02:53:56.183197 2193 master.cc:559] Master@127.2.36.125:43219 shutdown complete.
I20250115 02:53:56.195923 2193 master.cc:537] Master@127.2.36.124:44903 shutting down...
I20250115 02:53:56.211414 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:56.211912 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:56.212199 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P e800fe9800dc4ac8808518c32ff6c7aa: stopping tablet replica
I20250115 02:53:56.230150 2193 master.cc:559] Master@127.2.36.124:44903 shutdown complete.
[ OK ] RemoteKsckTest.TestTableFiltersNoMatch (3388 ms)
[ RUN ] RemoteKsckTest.TestTableFilters
I20250115 02:53:56.258147 2193 test_util.cc:274] Using random seed: -908048049
I20250115 02:53:56.259110 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:39225,127.2.36.125:35873,127.2.36.124:41281
I20250115 02:53:56.260360 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:56.265035 13322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:56.265566 13323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:56.266368 13325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:56.266916 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:56.267639 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:56.267818 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:56.267951 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909636267934 us; error 0 us; skew 500 ppm
I20250115 02:53:56.268428 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:56.270571 2193 webserver.cc:458] Webserver started at http://127.2.36.126:44599/ using document root <none> and password file <none>
I20250115 02:53:56.271102 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:56.271287 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:56.271517 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:56.272517 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "f7269d076a4c4c20ac3ae6117af75e1f"
format_stamp: "Formatted at 2025-01-15 02:53:56 on dist-test-slave-1lv2"
I20250115 02:53:56.276649 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:53:56.279657 13330 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:56.280401 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:56.280666 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "f7269d076a4c4c20ac3ae6117af75e1f"
format_stamp: "Formatted at 2025-01-15 02:53:56 on dist-test-slave-1lv2"
I20250115 02:53:56.280916 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:56.299906 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:56.300844 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:56.333814 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:39225
I20250115 02:53:56.333936 13381 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:39225 every 8 connection(s)
I20250115 02:53:56.337522 13382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:56.337667 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:56.343158 13384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:56.345450 13385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:56.346302 13382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:56.346946 13387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:56.347172 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:56.348323 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:56.348609 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:56.348841 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909636348820 us; error 0 us; skew 500 ppm
I20250115 02:53:56.349563 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:56.352241 2193 webserver.cc:458] Webserver started at http://127.2.36.125:45899/ using document root <none> and password file <none>
I20250115 02:53:56.352705 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:56.352874 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:56.353111 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:56.354092 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "417108afc1804786b50a49d58251d850"
format_stamp: "Formatted at 2025-01-15 02:53:56 on dist-test-slave-1lv2"
I20250115 02:53:56.357671 13382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:56.359141 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
W20250115 02:53:56.362811 13382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:35873: Network error: Client connection negotiation failed: client connection to 127.2.36.125:35873: connect: Connection refused (error 111)
I20250115 02:53:56.363408 13395 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:56.364105 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20250115 02:53:56.364373 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "417108afc1804786b50a49d58251d850"
format_stamp: "Formatted at 2025-01-15 02:53:56 on dist-test-slave-1lv2"
I20250115 02:53:56.364653 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:56.387964 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:56.388912 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:56.413322 13382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } attempt: 1
W20250115 02:53:56.419431 13382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:35873: Network error: Client connection negotiation failed: client connection to 127.2.36.125:35873: connect: Connection refused (error 111)
I20250115 02:53:56.425892 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:35873
I20250115 02:53:56.425971 13446 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:35873 every 8 connection(s)
I20250115 02:53:56.429481 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:56.429548 13447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 02:53:56.434904 13449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:56.435113 13450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:56.437263 13452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:56.437613 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:56.437876 13447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:56.438545 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:56.438795 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:56.438961 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909636438943 us; error 0 us; skew 500 ppm
I20250115 02:53:56.439668 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:56.442463 2193 webserver.cc:458] Webserver started at http://127.2.36.124:43649/ using document root <none> and password file <none>
I20250115 02:53:56.443006 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:56.443185 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:56.443425 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:56.444402 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "984f16f63d3d406a96364e1576789128"
format_stamp: "Formatted at 2025-01-15 02:53:56 on dist-test-slave-1lv2"
I20250115 02:53:56.448459 13447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:56.449337 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250115 02:53:56.453632 13459 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:56.454442 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:53:56.454720 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "984f16f63d3d406a96364e1576789128"
format_stamp: "Formatted at 2025-01-15 02:53:56 on dist-test-slave-1lv2"
I20250115 02:53:56.455031 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:56.458959 13382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } attempt: 2
I20250115 02:53:56.459833 13447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:56.464668 13447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:41281: Network error: Client connection negotiation failed: client connection to 127.2.36.124:41281: connect: Connection refused (error 111)
I20250115 02:53:56.471813 13382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:53:56.475700 13382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:41281: Network error: Client connection negotiation failed: client connection to 127.2.36.124:41281: connect: Connection refused (error 111)
I20250115 02:53:56.476423 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:56.477420 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:56.508143 13447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } attempt: 1
I20250115 02:53:56.512616 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:41281
I20250115 02:53:56.512701 13511 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:41281 every 8 connection(s)
I20250115 02:53:56.515928 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:53:56.516999 13513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:56.524966 13513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:56.530094 13447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850: Bootstrap starting.
I20250115 02:53:56.534972 13447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:56.535813 13513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:56.540652 13447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850: No bootstrap required, opened a new log
I20250115 02:53:56.541088 13382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } attempt: 1
I20250115 02:53:56.543498 13447 raft_consensus.cc:357] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } }
I20250115 02:53:56.544452 13447 raft_consensus.cc:383] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:56.544780 13447 raft_consensus.cc:738] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417108afc1804786b50a49d58251d850, State: Initialized, Role: FOLLOWER
I20250115 02:53:56.545616 13447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } }
I20250115 02:53:56.547798 13517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } } }
I20250115 02:53:56.548674 13513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:53:56.549010 13517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:56.549599 13447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:56.563004 13382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f: Bootstrap starting.
I20250115 02:53:56.569516 13382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:56.573616 13513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128: Bootstrap starting.
I20250115 02:53:56.574998 13382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f: No bootstrap required, opened a new log
I20250115 02:53:56.577924 13382 raft_consensus.cc:357] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } }
I20250115 02:53:56.578786 13382 raft_consensus.cc:383] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:56.579144 13382 raft_consensus.cc:738] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7269d076a4c4c20ac3ae6117af75e1f, State: Initialized, Role: FOLLOWER
I20250115 02:53:56.580060 13513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:56.579953 13382 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } }
I20250115 02:53:56.583544 13531 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } } }
I20250115 02:53:56.584404 13531 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:56.585489 13382 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [sys.catalog]: configured and running, proceeding with master startup.
W20250115 02:53:56.585534 13530 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:56.586126 13530 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:56.588040 13513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128: No bootstrap required, opened a new log
I20250115 02:53:56.591099 13513 raft_consensus.cc:357] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } }
I20250115 02:53:56.591867 13513 raft_consensus.cc:383] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:56.592157 13513 raft_consensus.cc:738] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 984f16f63d3d406a96364e1576789128, State: Initialized, Role: FOLLOWER
I20250115 02:53:56.592861 13513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } }
I20250115 02:53:56.595118 13539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } } }
I20250115 02:53:56.595916 13539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:56.597043 13513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:53:56.598596 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:53:56.598919 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250115 02:53:56.601579 13544 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:56.601890 13544 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:53:56.609616 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:56.612133 13555 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:53:56.612494 13555 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:53:56.616498 13556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:56.617200 13539 raft_consensus.cc:491] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 02:53:56.617715 13557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:56.617775 13539 raft_consensus.cc:513] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } }
W20250115 02:53:56.618867 13559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:56.619170 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:56.620043 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:56.620307 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:56.620517 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909636620497 us; error 0 us; skew 500 ppm
I20250115 02:53:56.620798 13539 leader_election.cc:290] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7269d076a4c4c20ac3ae6117af75e1f (127.2.36.126:39225), 417108afc1804786b50a49d58251d850 (127.2.36.125:35873)
I20250115 02:53:56.621400 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:56.620836 13357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "984f16f63d3d406a96364e1576789128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" is_pre_election: true
I20250115 02:53:56.620954 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "984f16f63d3d406a96364e1576789128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417108afc1804786b50a49d58251d850" is_pre_election: true
I20250115 02:53:56.622037 13357 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 984f16f63d3d406a96364e1576789128 in term 0.
I20250115 02:53:56.622128 13422 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 984f16f63d3d406a96364e1576789128 in term 0.
I20250115 02:53:56.623291 13460 leader_election.cc:304] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 984f16f63d3d406a96364e1576789128, f7269d076a4c4c20ac3ae6117af75e1f; no voters:
I20250115 02:53:56.624078 13539 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:56.624464 13539 raft_consensus.cc:491] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:56.624640 2193 webserver.cc:458] Webserver started at http://127.2.36.65:36945/ using document root <none> and password file <none>
I20250115 02:53:56.624766 13539 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:56.625180 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:56.625394 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:56.625630 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:56.626597 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "bf9b64f185d94f3b9e3bf797f730f325"
format_stamp: "Formatted at 2025-01-15 02:53:56 on dist-test-slave-1lv2"
I20250115 02:53:56.629578 13539 raft_consensus.cc:513] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } }
I20250115 02:53:56.630959 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250115 02:53:56.631124 13539 leader_election.cc:290] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [CANDIDATE]: Term 1 election: Requested vote from peers f7269d076a4c4c20ac3ae6117af75e1f (127.2.36.126:39225), 417108afc1804786b50a49d58251d850 (127.2.36.125:35873)
I20250115 02:53:56.631964 13357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "984f16f63d3d406a96364e1576789128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7269d076a4c4c20ac3ae6117af75e1f"
I20250115 02:53:56.632097 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "984f16f63d3d406a96364e1576789128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417108afc1804786b50a49d58251d850"
I20250115 02:53:56.632448 13357 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:56.632711 13422 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:56.635578 13564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:56.636543 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250115 02:53:56.636893 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "bf9b64f185d94f3b9e3bf797f730f325"
format_stamp: "Formatted at 2025-01-15 02:53:56 on dist-test-slave-1lv2"
I20250115 02:53:56.637234 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:56.638161 13357 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 984f16f63d3d406a96364e1576789128 in term 1.
I20250115 02:53:56.639156 13460 leader_election.cc:304] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 984f16f63d3d406a96364e1576789128, f7269d076a4c4c20ac3ae6117af75e1f; no voters:
I20250115 02:53:56.639884 13539 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:56.640194 13422 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 984f16f63d3d406a96364e1576789128 in term 1.
I20250115 02:53:56.641428 13539 raft_consensus.cc:695] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 1 LEADER]: Becoming Leader. State: Replica: 984f16f63d3d406a96364e1576789128, State: Running, Role: LEADER
I20250115 02:53:56.642375 13539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } }
I20250115 02:53:56.647428 13566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 984f16f63d3d406a96364e1576789128. Latest consensus state: current_term: 1 leader_uuid: "984f16f63d3d406a96364e1576789128" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } } }
I20250115 02:53:56.648150 13566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:56.649585 13571 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:53:56.651703 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:56.652779 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:56.654251 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:56.654403 13571 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:53:56.656783 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:56.657069 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250115 02:53:56.657290 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:56.657449 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:56.667914 13357 raft_consensus.cc:1270] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [term 1 FOLLOWER]: Refusing update from remote peer 984f16f63d3d406a96364e1576789128: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:56.669592 13566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:56.669716 13422 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [term 1 FOLLOWER]: Refusing update from remote peer 984f16f63d3d406a96364e1576789128: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:53:56.671538 13566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:56.680501 13531 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 984f16f63d3d406a96364e1576789128. Latest consensus state: current_term: 1 leader_uuid: "984f16f63d3d406a96364e1576789128" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } } }
I20250115 02:53:56.681286 13531 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:56.684507 13517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 984f16f63d3d406a96364e1576789128. Latest consensus state: current_term: 1 leader_uuid: "984f16f63d3d406a96364e1576789128" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } } }
I20250115 02:53:56.685386 13517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:56.687000 13539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "984f16f63d3d406a96364e1576789128" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } } }
I20250115 02:53:56.687750 13539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:56.689108 13539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "984f16f63d3d406a96364e1576789128" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } } }
I20250115 02:53:56.689635 13539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [sys.catalog]: This master's current role is: LEADER
I20250115 02:53:56.690716 13571 catalog_manager.cc:1348] Generated new cluster ID: 2c8c17f53c694782a63fd58d162027cb
I20250115 02:53:56.691103 13571 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:53:56.693315 13517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "984f16f63d3d406a96364e1576789128" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } } }
I20250115 02:53:56.694077 13517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:56.699733 13531 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "984f16f63d3d406a96364e1576789128" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7269d076a4c4c20ac3ae6117af75e1f" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 39225 } } peers { permanent_uuid: "417108afc1804786b50a49d58251d850" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 35873 } } peers { permanent_uuid: "984f16f63d3d406a96364e1576789128" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 41281 } } }
I20250115 02:53:56.700675 13531 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:53:56.734603 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:41449
I20250115 02:53:56.734717 13637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:41449 every 8 connection(s)
I20250115 02:53:56.744853 13571 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:53:56.746376 13571 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:53:56.769479 13571 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128: Generated new TSK 0
I20250115 02:53:56.770249 13571 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:53:56.781304 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:53:56.787397 13638 heartbeater.cc:346] Connected to a master server at 127.2.36.124:41281
I20250115 02:53:56.787920 13638 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:56.788808 13638 heartbeater.cc:509] Master 127.2.36.124:41281 requested a full tablet report, sending...
I20250115 02:53:56.791466 13477 ts_manager.cc:194] Registered new tserver with Master: bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65:41449)
W20250115 02:53:56.791698 13645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:56.793761 13646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:56.794639 13477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45620
I20250115 02:53:56.798236 13641 heartbeater.cc:346] Connected to a master server at 127.2.36.126:39225
I20250115 02:53:56.798683 13641 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:56.798707 13639 heartbeater.cc:346] Connected to a master server at 127.2.36.125:35873
I20250115 02:53:56.799229 13639 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:56.799381 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:56.799533 13641 heartbeater.cc:509] Master 127.2.36.126:39225 requested a full tablet report, sending...
I20250115 02:53:56.799907 13639 heartbeater.cc:509] Master 127.2.36.125:35873 requested a full tablet report, sending...
W20250115 02:53:56.801107 13648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:56.802417 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:56.802651 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:56.802850 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909636802834 us; error 0 us; skew 500 ppm
I20250115 02:53:56.802915 13412 ts_manager.cc:194] Registered new tserver with Master: bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65:41449)
I20250115 02:53:56.803146 13347 ts_manager.cc:194] Registered new tserver with Master: bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65:41449)
I20250115 02:53:56.803511 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:56.806582 2193 webserver.cc:458] Webserver started at http://127.2.36.66:46051/ using document root <none> and password file <none>
I20250115 02:53:56.807035 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:56.807192 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:56.807399 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:56.808198 13638 heartbeater.cc:501] Master 127.2.36.124:41281 was elected leader, sending a full tablet report...
I20250115 02:53:56.808339 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "14e267b06fb54672bcfbbd9a7ea61b76"
format_stamp: "Formatted at 2025-01-15 02:53:56 on dist-test-slave-1lv2"
I20250115 02:53:56.812829 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250115 02:53:56.816387 13653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:56.817265 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250115 02:53:56.817616 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "14e267b06fb54672bcfbbd9a7ea61b76"
format_stamp: "Formatted at 2025-01-15 02:53:56 on dist-test-slave-1lv2"
I20250115 02:53:56.817975 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:56.838482 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:56.839668 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:56.840977 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:56.843550 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:56.843760 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:56.843983 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:56.844134 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:56.887315 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:42909
I20250115 02:53:56.887421 13715 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:42909 every 8 connection(s)
I20250115 02:53:56.894963 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:53:56.915792 13724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:56.921880 13723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:53:56.925992 13727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:53:56.926659 2193 server_base.cc:1034] running on GCE node
I20250115 02:53:56.928078 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:53:56.928464 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:53:56.928766 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909636928749 us; error 0 us; skew 500 ppm
I20250115 02:53:56.928828 13718 heartbeater.cc:346] Connected to a master server at 127.2.36.126:39225
I20250115 02:53:56.929344 13718 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:56.929708 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:53:56.930292 13718 heartbeater.cc:509] Master 127.2.36.126:39225 requested a full tablet report, sending...
I20250115 02:53:56.931603 13717 heartbeater.cc:346] Connected to a master server at 127.2.36.125:35873
I20250115 02:53:56.932077 13717 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:56.932994 13717 heartbeater.cc:509] Master 127.2.36.125:35873 requested a full tablet report, sending...
I20250115 02:53:56.933598 13347 ts_manager.cc:194] Registered new tserver with Master: 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909)
I20250115 02:53:56.936039 13412 ts_manager.cc:194] Registered new tserver with Master: 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909)
I20250115 02:53:56.941113 2193 webserver.cc:458] Webserver started at http://127.2.36.67:37487/ using document root <none> and password file <none>
I20250115 02:53:56.941373 13716 heartbeater.cc:346] Connected to a master server at 127.2.36.124:41281
I20250115 02:53:56.941733 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:53:56.941766 13716 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:56.942104 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:53:56.942406 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:53:56.942653 13716 heartbeater.cc:509] Master 127.2.36.124:41281 requested a full tablet report, sending...
I20250115 02:53:56.943497 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "5443d586ce28493ab00dda0282279b84"
format_stamp: "Formatted at 2025-01-15 02:53:56 on dist-test-slave-1lv2"
I20250115 02:53:56.944595 13477 ts_manager.cc:194] Registered new tserver with Master: 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909)
I20250115 02:53:56.946372 13477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45630
I20250115 02:53:56.952396 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20250115 02:53:56.955986 13731 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:56.956784 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20250115 02:53:56.957077 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "5443d586ce28493ab00dda0282279b84"
format_stamp: "Formatted at 2025-01-15 02:53:56 on dist-test-slave-1lv2"
I20250115 02:53:56.957351 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:53:56.959923 13716 heartbeater.cc:501] Master 127.2.36.124:41281 was elected leader, sending a full tablet report...
I20250115 02:53:56.976365 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:53:56.977931 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:53:56.980163 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:53:56.982570 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:53:56.982825 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250115 02:53:56.983047 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:53:56.983180 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:53:57.048491 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:42515
I20250115 02:53:57.048583 13793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:42515 every 8 connection(s)
I20250115 02:53:57.068226 13794 heartbeater.cc:346] Connected to a master server at 127.2.36.124:41281
I20250115 02:53:57.068711 13794 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:57.069588 13794 heartbeater.cc:509] Master 127.2.36.124:41281 requested a full tablet report, sending...
I20250115 02:53:57.071945 13477 ts_manager.cc:194] Registered new tserver with Master: 5443d586ce28493ab00dda0282279b84 (127.2.36.67:42515)
I20250115 02:53:57.073941 13477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45632
I20250115 02:53:57.087643 13795 heartbeater.cc:346] Connected to a master server at 127.2.36.125:35873
I20250115 02:53:57.088091 13795 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:57.088774 13795 heartbeater.cc:509] Master 127.2.36.125:35873 requested a full tablet report, sending...
I20250115 02:53:57.094512 13794 heartbeater.cc:501] Master 127.2.36.124:41281 was elected leader, sending a full tablet report...
I20250115 02:53:57.096994 13412 ts_manager.cc:194] Registered new tserver with Master: 5443d586ce28493ab00dda0282279b84 (127.2.36.67:42515)
I20250115 02:53:57.097718 13797 heartbeater.cc:346] Connected to a master server at 127.2.36.126:39225
I20250115 02:53:57.098143 13797 heartbeater.cc:463] Registering TS with master...
I20250115 02:53:57.098825 13797 heartbeater.cc:509] Master 127.2.36.126:39225 requested a full tablet report, sending...
I20250115 02:53:57.101342 13347 ts_manager.cc:194] Registered new tserver with Master: 5443d586ce28493ab00dda0282279b84 (127.2.36.67:42515)
I20250115 02:53:57.104312 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.000678817s
I20250115 02:53:57.157176 13477 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45636:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:53:57.159091 13477 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:53:57.285017 13603 tablet_service.cc:1467] Processing CreateTablet for tablet 9bf81025b11c42baa1536bb073019017 (DEFAULT_TABLE table=ksck-test-table [id=3379b24e52cf47ea9ccbb25b24f36ff1]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:57.286168 13603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf81025b11c42baa1536bb073019017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:57.287761 13601 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3686c295604093be1507695d8c33e7 (DEFAULT_TABLE table=ksck-test-table [id=3379b24e52cf47ea9ccbb25b24f36ff1]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:57.289124 13601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a3686c295604093be1507695d8c33e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:57.291538 13602 tablet_service.cc:1467] Processing CreateTablet for tablet 96379a1087a24734951ec73f750631fb (DEFAULT_TABLE table=ksck-test-table [id=3379b24e52cf47ea9ccbb25b24f36ff1]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:57.292562 13602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96379a1087a24734951ec73f750631fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:57.294616 13759 tablet_service.cc:1467] Processing CreateTablet for tablet 9bf81025b11c42baa1536bb073019017 (DEFAULT_TABLE table=ksck-test-table [id=3379b24e52cf47ea9ccbb25b24f36ff1]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:57.295981 13759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf81025b11c42baa1536bb073019017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:57.296391 13758 tablet_service.cc:1467] Processing CreateTablet for tablet 96379a1087a24734951ec73f750631fb (DEFAULT_TABLE table=ksck-test-table [id=3379b24e52cf47ea9ccbb25b24f36ff1]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:57.297758 13758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96379a1087a24734951ec73f750631fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:57.302115 13757 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3686c295604093be1507695d8c33e7 (DEFAULT_TABLE table=ksck-test-table [id=3379b24e52cf47ea9ccbb25b24f36ff1]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:57.303627 13757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a3686c295604093be1507695d8c33e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:57.332506 13815 tablet_bootstrap.cc:492] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325: Bootstrap starting.
I20250115 02:53:57.349829 13816 tablet_bootstrap.cc:492] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84: Bootstrap starting.
I20250115 02:53:57.354379 13679 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3686c295604093be1507695d8c33e7 (DEFAULT_TABLE table=ksck-test-table [id=3379b24e52cf47ea9ccbb25b24f36ff1]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:53:57.355754 13679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a3686c295604093be1507695d8c33e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:57.360080 13816 tablet_bootstrap.cc:654] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:57.351753 13681 tablet_service.cc:1467] Processing CreateTablet for tablet 9bf81025b11c42baa1536bb073019017 (DEFAULT_TABLE table=ksck-test-table [id=3379b24e52cf47ea9ccbb25b24f36ff1]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:53:57.363914 13681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf81025b11c42baa1536bb073019017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:57.352455 13815 tablet_bootstrap.cc:654] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:57.353025 13680 tablet_service.cc:1467] Processing CreateTablet for tablet 96379a1087a24734951ec73f750631fb (DEFAULT_TABLE table=ksck-test-table [id=3379b24e52cf47ea9ccbb25b24f36ff1]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:53:57.367874 13680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96379a1087a24734951ec73f750631fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:57.371703 13816 tablet_bootstrap.cc:492] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84: No bootstrap required, opened a new log
I20250115 02:53:57.372140 13816 ts_tablet_manager.cc:1397] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.000s
I20250115 02:53:57.374747 13816 raft_consensus.cc:357] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.375635 13816 raft_consensus.cc:383] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:57.376040 13816 raft_consensus.cc:738] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5443d586ce28493ab00dda0282279b84, State: Initialized, Role: FOLLOWER
I20250115 02:53:57.376911 13816 consensus_queue.cc:260] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.431434 13816 ts_tablet_manager.cc:1428] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84: Time spent starting tablet: real 0.059s user 0.024s sys 0.017s
I20250115 02:53:57.465206 13816 tablet_bootstrap.cc:492] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84: Bootstrap starting.
I20250115 02:53:57.467170 13820 tablet_bootstrap.cc:492] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76: Bootstrap starting.
I20250115 02:53:57.468904 13815 tablet_bootstrap.cc:492] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325: No bootstrap required, opened a new log
I20250115 02:53:57.469494 13815 ts_tablet_manager.cc:1397] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325: Time spent bootstrapping tablet: real 0.137s user 0.038s sys 0.035s
I20250115 02:53:57.471617 13816 tablet_bootstrap.cc:654] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:57.472333 13815 raft_consensus.cc:357] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.478106 13815 raft_consensus.cc:383] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:57.478461 13815 raft_consensus.cc:738] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9b64f185d94f3b9e3bf797f730f325, State: Initialized, Role: FOLLOWER
I20250115 02:53:57.479221 13815 consensus_queue.cc:260] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.482348 13815 ts_tablet_manager.cc:1428] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325: Time spent starting tablet: real 0.013s user 0.009s sys 0.001s
I20250115 02:53:57.484215 13815 tablet_bootstrap.cc:492] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325: Bootstrap starting.
I20250115 02:53:57.494482 13815 tablet_bootstrap.cc:654] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:57.498486 13820 tablet_bootstrap.cc:654] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:57.506525 13820 tablet_bootstrap.cc:492] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76: No bootstrap required, opened a new log
I20250115 02:53:57.507036 13820 ts_tablet_manager.cc:1397] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.006s
I20250115 02:53:57.509742 13820 raft_consensus.cc:357] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.511755 13820 raft_consensus.cc:383] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:57.512086 13820 raft_consensus.cc:738] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e267b06fb54672bcfbbd9a7ea61b76, State: Initialized, Role: FOLLOWER
I20250115 02:53:57.513059 13815 tablet_bootstrap.cc:492] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325: No bootstrap required, opened a new log
I20250115 02:53:57.512943 13820 consensus_queue.cc:260] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.513654 13815 ts_tablet_manager.cc:1397] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.009s
I20250115 02:53:57.516579 13815 raft_consensus.cc:357] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.517342 13815 raft_consensus.cc:383] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:57.517663 13815 raft_consensus.cc:738] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9b64f185d94f3b9e3bf797f730f325, State: Initialized, Role: FOLLOWER
I20250115 02:53:57.518427 13815 consensus_queue.cc:260] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.520813 13815 ts_tablet_manager.cc:1428] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250115 02:53:57.528373 13815 tablet_bootstrap.cc:492] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325: Bootstrap starting.
I20250115 02:53:57.535308 13815 tablet_bootstrap.cc:654] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:57.545507 13816 tablet_bootstrap.cc:492] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84: No bootstrap required, opened a new log
I20250115 02:53:57.546044 13816 ts_tablet_manager.cc:1397] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84: Time spent bootstrapping tablet: real 0.081s user 0.019s sys 0.000s
I20250115 02:53:57.548867 13816 raft_consensus.cc:357] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.549626 13816 raft_consensus.cc:383] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:57.549902 13816 raft_consensus.cc:738] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5443d586ce28493ab00dda0282279b84, State: Initialized, Role: FOLLOWER
I20250115 02:53:57.550271 13815 tablet_bootstrap.cc:492] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325: No bootstrap required, opened a new log
I20250115 02:53:57.551016 13815 ts_tablet_manager.cc:1397] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.009s
I20250115 02:53:57.550827 13816 consensus_queue.cc:260] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.553356 13816 ts_tablet_manager.cc:1428] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250115 02:53:57.555027 13820 ts_tablet_manager.cc:1428] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76: Time spent starting tablet: real 0.048s user 0.007s sys 0.025s
I20250115 02:53:57.555992 13815 raft_consensus.cc:357] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.556915 13815 raft_consensus.cc:383] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:57.556021 13816 tablet_bootstrap.cc:492] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84: Bootstrap starting.
I20250115 02:53:57.557327 13815 raft_consensus.cc:738] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9b64f185d94f3b9e3bf797f730f325, State: Initialized, Role: FOLLOWER
I20250115 02:53:57.558990 13815 consensus_queue.cc:260] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.563506 13816 tablet_bootstrap.cc:654] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:57.557077 13820 tablet_bootstrap.cc:492] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76: Bootstrap starting.
I20250115 02:53:57.568816 13815 ts_tablet_manager.cc:1428] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325: Time spent starting tablet: real 0.017s user 0.012s sys 0.000s
I20250115 02:53:57.569317 13820 tablet_bootstrap.cc:654] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:57.571615 13819 raft_consensus.cc:491] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:57.572131 13819 raft_consensus.cc:513] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.574542 13819 leader_election.cc:290] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65:41449), 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909)
I20250115 02:53:57.585690 13613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3686c295604093be1507695d8c33e7" candidate_uuid: "5443d586ce28493ab00dda0282279b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9b64f185d94f3b9e3bf797f730f325" is_pre_election: true
I20250115 02:53:57.586494 13613 raft_consensus.cc:2463] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5443d586ce28493ab00dda0282279b84 in term 0.
I20250115 02:53:57.591058 13733 leader_election.cc:304] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5443d586ce28493ab00dda0282279b84, bf9b64f185d94f3b9e3bf797f730f325; no voters:
I20250115 02:53:57.591871 13530 catalog_manager.cc:1260] Loaded cluster ID: 2c8c17f53c694782a63fd58d162027cb
I20250115 02:53:57.592027 13819 raft_consensus.cc:2798] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:57.592391 13819 raft_consensus.cc:491] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:57.592738 13819 raft_consensus.cc:3054] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:57.594300 13530 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850: loading cluster ID for follower catalog manager: success
I20250115 02:53:57.598273 13827 raft_consensus.cc:491] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:57.601717 13816 tablet_bootstrap.cc:492] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84: No bootstrap required, opened a new log
I20250115 02:53:57.601919 13827 raft_consensus.cc:513] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.604646 13816 ts_tablet_manager.cc:1397] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.004s
I20250115 02:53:57.607435 13816 raft_consensus.cc:357] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.604203 13820 tablet_bootstrap.cc:492] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76: No bootstrap required, opened a new log
I20250115 02:53:57.608790 13820 ts_tablet_manager.cc:1397] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76: Time spent bootstrapping tablet: real 0.052s user 0.014s sys 0.000s
I20250115 02:53:57.614431 13544 catalog_manager.cc:1260] Loaded cluster ID: 2c8c17f53c694782a63fd58d162027cb
I20250115 02:53:57.614917 13544 catalog_manager.cc:1553] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f: loading cluster ID for follower catalog manager: success
I20250115 02:53:57.616681 13530 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850: acquiring CA information for follower catalog manager: success
I20250115 02:53:57.617069 13820 raft_consensus.cc:357] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.631165 13820 raft_consensus.cc:383] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:57.631543 13820 raft_consensus.cc:738] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e267b06fb54672bcfbbd9a7ea61b76, State: Initialized, Role: FOLLOWER
I20250115 02:53:57.617805 13816 raft_consensus.cc:383] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:57.624948 13691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3686c295604093be1507695d8c33e7" candidate_uuid: "5443d586ce28493ab00dda0282279b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" is_pre_election: true
I20250115 02:53:57.632341 13820 consensus_queue.cc:260] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.633431 13691 raft_consensus.cc:2463] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5443d586ce28493ab00dda0282279b84 in term 0.
I20250115 02:53:57.634738 13820 ts_tablet_manager.cc:1428] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76: Time spent starting tablet: real 0.026s user 0.004s sys 0.002s
I20250115 02:53:57.635803 13820 tablet_bootstrap.cc:492] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76: Bootstrap starting.
I20250115 02:53:57.633221 13816 raft_consensus.cc:738] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5443d586ce28493ab00dda0282279b84, State: Initialized, Role: FOLLOWER
I20250115 02:53:57.630053 13819 raft_consensus.cc:513] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.653489 13819 leader_election.cc:290] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 election: Requested vote from peers bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65:41449), 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909)
I20250115 02:53:57.653487 13530 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:57.656982 13691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3686c295604093be1507695d8c33e7" candidate_uuid: "5443d586ce28493ab00dda0282279b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e267b06fb54672bcfbbd9a7ea61b76"
I20250115 02:53:57.657754 13691 raft_consensus.cc:3054] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:57.660439 13613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3686c295604093be1507695d8c33e7" candidate_uuid: "5443d586ce28493ab00dda0282279b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9b64f185d94f3b9e3bf797f730f325"
I20250115 02:53:57.661147 13613 raft_consensus.cc:3054] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:57.665285 13691 raft_consensus.cc:2463] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5443d586ce28493ab00dda0282279b84 in term 1.
I20250115 02:53:57.668823 13613 raft_consensus.cc:2463] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5443d586ce28493ab00dda0282279b84 in term 1.
I20250115 02:53:57.653928 13816 consensus_queue.cc:260] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.674050 13733 leader_election.cc:304] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e267b06fb54672bcfbbd9a7ea61b76, 5443d586ce28493ab00dda0282279b84; no voters:
I20250115 02:53:57.649591 13544 catalog_manager.cc:1575] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f: acquiring CA information for follower catalog manager: success
I20250115 02:53:57.676934 13819 raft_consensus.cc:2798] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:57.627904 13827 leader_election.cc:290] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909), 5443d586ce28493ab00dda0282279b84 (127.2.36.67:42515)
I20250115 02:53:57.679826 13816 ts_tablet_manager.cc:1428] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84: Time spent starting tablet: real 0.075s user 0.007s sys 0.000s
I20250115 02:53:57.689378 13544 catalog_manager.cc:1603] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:53:57.694778 13819 raft_consensus.cc:695] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 1 LEADER]: Becoming Leader. State: Replica: 5443d586ce28493ab00dda0282279b84, State: Running, Role: LEADER
I20250115 02:53:57.695307 13833 raft_consensus.cc:491] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:57.695816 13819 consensus_queue.cc:237] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.704402 13820 tablet_bootstrap.cc:654] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:57.695801 13833 raft_consensus.cc:513] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.708494 13833 leader_election.cc:290] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65:41449), 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909)
I20250115 02:53:57.718195 13691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf81025b11c42baa1536bb073019017" candidate_uuid: "5443d586ce28493ab00dda0282279b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" is_pre_election: true
W20250115 02:53:57.719764 13733 leader_election.cc:343] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:57.718631 13613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf81025b11c42baa1536bb073019017" candidate_uuid: "5443d586ce28493ab00dda0282279b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9b64f185d94f3b9e3bf797f730f325" is_pre_election: true
I20250115 02:53:57.724574 13691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96379a1087a24734951ec73f750631fb" candidate_uuid: "bf9b64f185d94f3b9e3bf797f730f325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" is_pre_election: true
I20250115 02:53:57.725255 13691 raft_consensus.cc:2463] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9b64f185d94f3b9e3bf797f730f325 in term 0.
I20250115 02:53:57.724534 13613 raft_consensus.cc:2463] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5443d586ce28493ab00dda0282279b84 in term 0.
I20250115 02:53:57.726955 13567 leader_election.cc:304] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e267b06fb54672bcfbbd9a7ea61b76, bf9b64f185d94f3b9e3bf797f730f325; no voters:
I20250115 02:53:57.727819 13827 raft_consensus.cc:2798] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:57.728245 13827 raft_consensus.cc:491] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:57.728602 13827 raft_consensus.cc:3054] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:57.728060 13733 leader_election.cc:304] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5443d586ce28493ab00dda0282279b84, bf9b64f185d94f3b9e3bf797f730f325; no voters: 14e267b06fb54672bcfbbd9a7ea61b76
I20250115 02:53:57.733320 13833 raft_consensus.cc:2798] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:57.733697 13833 raft_consensus.cc:491] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:57.734011 13833 raft_consensus.cc:3054] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:57.743913 13769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96379a1087a24734951ec73f750631fb" candidate_uuid: "bf9b64f185d94f3b9e3bf797f730f325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5443d586ce28493ab00dda0282279b84" is_pre_election: true
I20250115 02:53:57.744652 13769 raft_consensus.cc:2463] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9b64f185d94f3b9e3bf797f730f325 in term 0.
I20250115 02:53:57.745764 13827 raft_consensus.cc:513] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.749547 13691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96379a1087a24734951ec73f750631fb" candidate_uuid: "bf9b64f185d94f3b9e3bf797f730f325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e267b06fb54672bcfbbd9a7ea61b76"
I20250115 02:53:57.750125 13691 raft_consensus.cc:3054] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:57.751467 13769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96379a1087a24734951ec73f750631fb" candidate_uuid: "bf9b64f185d94f3b9e3bf797f730f325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5443d586ce28493ab00dda0282279b84"
I20250115 02:53:57.752096 13769 raft_consensus.cc:3054] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:57.759646 13827 leader_election.cc:290] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [CANDIDATE]: Term 1 election: Requested vote from peers 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909), 5443d586ce28493ab00dda0282279b84 (127.2.36.67:42515)
I20250115 02:53:57.760192 13833 raft_consensus.cc:513] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.760607 13769 raft_consensus.cc:2463] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9b64f185d94f3b9e3bf797f730f325 in term 1.
I20250115 02:53:57.762977 13568 leader_election.cc:304] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5443d586ce28493ab00dda0282279b84, bf9b64f185d94f3b9e3bf797f730f325; no voters:
I20250115 02:53:57.763427 13833 leader_election.cc:290] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 election: Requested vote from peers bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65:41449), 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909)
I20250115 02:53:57.760234 13691 raft_consensus.cc:2463] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9b64f185d94f3b9e3bf797f730f325 in term 1.
I20250115 02:53:57.751554 13476 catalog_manager.cc:5526] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5443d586ce28493ab00dda0282279b84 (127.2.36.67). New cstate: current_term: 1 leader_uuid: "5443d586ce28493ab00dda0282279b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } health_report { overall_health: HEALTHY } } }
I20250115 02:53:57.764526 13827 raft_consensus.cc:2798] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:57.771026 13827 raft_consensus.cc:695] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 1 LEADER]: Becoming Leader. State: Replica: bf9b64f185d94f3b9e3bf797f730f325, State: Running, Role: LEADER
I20250115 02:53:57.771950 13827 consensus_queue.cc:237] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.761310 13820 tablet_bootstrap.cc:492] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76: No bootstrap required, opened a new log
I20250115 02:53:57.764840 13613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf81025b11c42baa1536bb073019017" candidate_uuid: "5443d586ce28493ab00dda0282279b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9b64f185d94f3b9e3bf797f730f325"
I20250115 02:53:57.775137 13613 raft_consensus.cc:3054] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:57.765622 13690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf81025b11c42baa1536bb073019017" candidate_uuid: "5443d586ce28493ab00dda0282279b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e267b06fb54672bcfbbd9a7ea61b76"
W20250115 02:53:57.779929 13733 leader_election.cc:343] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:53:57.780534 13477 catalog_manager.cc:5526] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "bf9b64f185d94f3b9e3bf797f730f325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20250115 02:53:57.790601 13613 raft_consensus.cc:2463] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5443d586ce28493ab00dda0282279b84 in term 1.
I20250115 02:53:57.793138 13733 leader_election.cc:304] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5443d586ce28493ab00dda0282279b84, bf9b64f185d94f3b9e3bf797f730f325; no voters: 14e267b06fb54672bcfbbd9a7ea61b76
I20250115 02:53:57.794696 13833 raft_consensus.cc:2798] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:57.795306 13833 raft_consensus.cc:695] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 1 LEADER]: Becoming Leader. State: Replica: 5443d586ce28493ab00dda0282279b84, State: Running, Role: LEADER
I20250115 02:53:57.796361 13833 consensus_queue.cc:237] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.791422 13820 ts_tablet_manager.cc:1397] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76: Time spent bootstrapping tablet: real 0.156s user 0.010s sys 0.011s
I20250115 02:53:57.803959 13820 raft_consensus.cc:357] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.815665 13820 raft_consensus.cc:383] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:57.816000 13820 raft_consensus.cc:738] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e267b06fb54672bcfbbd9a7ea61b76, State: Initialized, Role: FOLLOWER
I20250115 02:53:57.816721 13820 consensus_queue.cc:260] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.819092 13820 ts_tablet_manager.cc:1428] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76: Time spent starting tablet: real 0.018s user 0.003s sys 0.003s
I20250115 02:53:57.842221 13475 catalog_manager.cc:5526] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5443d586ce28493ab00dda0282279b84 (127.2.36.67). New cstate: current_term: 1 leader_uuid: "5443d586ce28493ab00dda0282279b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } health_report { overall_health: HEALTHY } } }
I20250115 02:53:57.881472 13825 raft_consensus.cc:491] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:57.881983 13825 raft_consensus.cc:513] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:57.884382 13825 leader_election.cc:290] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65:41449), 5443d586ce28493ab00dda0282279b84 (127.2.36.67:42515)
I20250115 02:53:57.901749 13613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf81025b11c42baa1536bb073019017" candidate_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9b64f185d94f3b9e3bf797f730f325" is_pre_election: true
I20250115 02:53:57.902582 13613 raft_consensus.cc:2388] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 14e267b06fb54672bcfbbd9a7ea61b76 in current term 1: Already voted for candidate 5443d586ce28493ab00dda0282279b84 in this term.
I20250115 02:53:57.909214 13769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf81025b11c42baa1536bb073019017" candidate_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5443d586ce28493ab00dda0282279b84" is_pre_election: true
I20250115 02:53:57.910912 13656 leader_election.cc:304] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14e267b06fb54672bcfbbd9a7ea61b76; no voters: 5443d586ce28493ab00dda0282279b84, bf9b64f185d94f3b9e3bf797f730f325
I20250115 02:53:57.911974 13825 raft_consensus.cc:3054] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:57.918392 13825 raft_consensus.cc:2743] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:53:58.040849 13477 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45636:
name: "other-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "value"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\007\001d\000\000\000""\006\001\000\000\000\000\007\001d\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:53:58.042980 13477 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table other-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:53:58.104614 13602 tablet_service.cc:1467] Processing CreateTablet for tablet 5b63c3cae1104bd098c4dcea978faa04 (DEFAULT_TABLE table=other-table [id=c4435e41ca3a4543b823cdcbb898b664]), partition=RANGE (key) PARTITION 0 <= VALUES < 100
I20250115 02:53:58.105715 13602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b63c3cae1104bd098c4dcea978faa04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:58.120906 13680 tablet_service.cc:1467] Processing CreateTablet for tablet 5b63c3cae1104bd098c4dcea978faa04 (DEFAULT_TABLE table=other-table [id=c4435e41ca3a4543b823cdcbb898b664]), partition=RANGE (key) PARTITION 0 <= VALUES < 100
I20250115 02:53:58.122272 13680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b63c3cae1104bd098c4dcea978faa04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:58.123203 13757 tablet_service.cc:1467] Processing CreateTablet for tablet 5b63c3cae1104bd098c4dcea978faa04 (DEFAULT_TABLE table=other-table [id=c4435e41ca3a4543b823cdcbb898b664]), partition=RANGE (key) PARTITION 0 <= VALUES < 100
I20250115 02:53:58.124670 13757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b63c3cae1104bd098c4dcea978faa04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:53:58.148730 13816 tablet_bootstrap.cc:492] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84: Bootstrap starting.
I20250115 02:53:58.152294 13820 tablet_bootstrap.cc:492] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76: Bootstrap starting.
I20250115 02:53:58.154496 13816 tablet_bootstrap.cc:654] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:58.159474 13820 tablet_bootstrap.cc:654] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:58.173463 13827 consensus_queue.cc:1035] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:58.180439 13816 tablet_bootstrap.cc:492] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84: No bootstrap required, opened a new log
I20250115 02:53:58.181069 13816 ts_tablet_manager.cc:1397] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.003s
I20250115 02:53:58.184002 13816 raft_consensus.cc:357] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:58.184548 13816 raft_consensus.cc:383] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:58.184779 13816 raft_consensus.cc:738] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5443d586ce28493ab00dda0282279b84, State: Initialized, Role: FOLLOWER
I20250115 02:53:58.185480 13816 consensus_queue.cc:260] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:58.188719 13816 ts_tablet_manager.cc:1428] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250115 02:53:58.194379 13856 tablet_bootstrap.cc:492] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325: Bootstrap starting.
I20250115 02:53:58.200160 13820 tablet_bootstrap.cc:492] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76: No bootstrap required, opened a new log
I20250115 02:53:58.200568 13856 tablet_bootstrap.cc:654] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325: Neither blocks nor log segments found. Creating new log.
I20250115 02:53:58.219789 13827 consensus_queue.cc:1035] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:58.227183 13820 ts_tablet_manager.cc:1397] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76: Time spent bootstrapping tablet: real 0.075s user 0.019s sys 0.001s
I20250115 02:53:58.258339 13820 raft_consensus.cc:357] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:58.259549 13820 raft_consensus.cc:383] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:58.259963 13820 raft_consensus.cc:738] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e267b06fb54672bcfbbd9a7ea61b76, State: Initialized, Role: FOLLOWER
I20250115 02:53:58.260870 13820 consensus_queue.cc:260] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:58.276124 13856 tablet_bootstrap.cc:492] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325: No bootstrap required, opened a new log
I20250115 02:53:58.276777 13856 ts_tablet_manager.cc:1397] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325: Time spent bootstrapping tablet: real 0.083s user 0.024s sys 0.052s
I20250115 02:53:58.279482 13856 raft_consensus.cc:357] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:58.279731 13833 consensus_queue.cc:1035] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:58.280304 13856 raft_consensus.cc:383] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:53:58.280728 13856 raft_consensus.cc:738] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9b64f185d94f3b9e3bf797f730f325, State: Initialized, Role: FOLLOWER
I20250115 02:53:58.281567 13856 consensus_queue.cc:260] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:58.289371 13833 consensus_queue.cc:1035] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:58.294996 13856 ts_tablet_manager.cc:1428] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325: Time spent starting tablet: real 0.018s user 0.004s sys 0.006s
I20250115 02:53:58.303205 13820 ts_tablet_manager.cc:1428] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76: Time spent starting tablet: real 0.075s user 0.005s sys 0.002s
I20250115 02:53:58.322163 13833 raft_consensus.cc:491] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:53:58.322700 13833 raft_consensus.cc:513] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:58.325045 13833 leader_election.cc:290] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65:41449), 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909)
I20250115 02:53:58.327378 13690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b63c3cae1104bd098c4dcea978faa04" candidate_uuid: "5443d586ce28493ab00dda0282279b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" is_pre_election: true
I20250115 02:53:58.327456 13612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b63c3cae1104bd098c4dcea978faa04" candidate_uuid: "5443d586ce28493ab00dda0282279b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9b64f185d94f3b9e3bf797f730f325" is_pre_election: true
I20250115 02:53:58.328267 13690 raft_consensus.cc:2463] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5443d586ce28493ab00dda0282279b84 in term 0.
I20250115 02:53:58.328372 13612 raft_consensus.cc:2463] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5443d586ce28493ab00dda0282279b84 in term 0.
I20250115 02:53:58.329782 13733 leader_election.cc:304] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e267b06fb54672bcfbbd9a7ea61b76, 5443d586ce28493ab00dda0282279b84; no voters:
I20250115 02:53:58.331539 13867 raft_consensus.cc:2798] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:53:58.331914 13867 raft_consensus.cc:491] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:53:58.332222 13867 raft_consensus.cc:3054] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:58.344049 13867 raft_consensus.cc:513] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:58.346051 13867 leader_election.cc:290] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 election: Requested vote from peers bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65:41449), 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909)
I20250115 02:53:58.347432 13613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b63c3cae1104bd098c4dcea978faa04" candidate_uuid: "5443d586ce28493ab00dda0282279b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9b64f185d94f3b9e3bf797f730f325"
I20250115 02:53:58.347705 13690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b63c3cae1104bd098c4dcea978faa04" candidate_uuid: "5443d586ce28493ab00dda0282279b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e267b06fb54672bcfbbd9a7ea61b76"
I20250115 02:53:58.348170 13613 raft_consensus.cc:3054] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:58.348380 13690 raft_consensus.cc:3054] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:53:58.356783 13690 raft_consensus.cc:2463] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5443d586ce28493ab00dda0282279b84 in term 1.
I20250115 02:53:58.359338 13733 leader_election.cc:304] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e267b06fb54672bcfbbd9a7ea61b76, 5443d586ce28493ab00dda0282279b84; no voters:
I20250115 02:53:58.361778 13867 raft_consensus.cc:2798] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:53:58.362257 13867 raft_consensus.cc:695] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 1 LEADER]: Becoming Leader. State: Replica: 5443d586ce28493ab00dda0282279b84, State: Running, Role: LEADER
I20250115 02:53:58.363349 13867 consensus_queue.cc:237] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } }
I20250115 02:53:58.365888 13613 raft_consensus.cc:2463] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5443d586ce28493ab00dda0282279b84 in term 1.
I20250115 02:53:58.372994 13477 catalog_manager.cc:5526] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5443d586ce28493ab00dda0282279b84 (127.2.36.67). New cstate: current_term: 1 leader_uuid: "5443d586ce28493ab00dda0282279b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5443d586ce28493ab00dda0282279b84" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 42515 } health_report { overall_health: HEALTHY } } }
I20250115 02:53:58.400895 13867 consensus_queue.cc:1035] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9b64f185d94f3b9e3bf797f730f325" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 41449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:58.412922 13867 consensus_queue.cc:1035] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e267b06fb54672bcfbbd9a7ea61b76" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 42909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:53:58.413276 13475 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45636:
table { table_name: "other-table" } alter_schema_steps { type: DROP_RANGE_PARTITION drop_range_partition { range_bounds { rows: "\006\001\000\000\000\000\007\001d\000\000\000""\006\001\000\000\000\000\007\001d\000\000\000" indirect_data: """" } } } schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "value" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } modify_external_catalogs: true
I20250115 02:53:58.451856 13475 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128: Sending DeleteTablet for 3 replicas of tablet 5b63c3cae1104bd098c4dcea978faa04
I20250115 02:53:58.454614 13757 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b63c3cae1104bd098c4dcea978faa04 with delete_type TABLET_DATA_DELETED (Partition dropped at 2025-01-15 02:53:58 UTC) from {username='slave'} at 127.0.0.1:54622
I20250115 02:53:58.459446 13872 tablet_replica.cc:331] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84: stopping tablet replica
I20250115 02:53:58.460677 13872 raft_consensus.cc:2238] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:58.462584 13872 pending_rounds.cc:62] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84: Trying to abort 1 pending ops.
I20250115 02:53:58.463757 13872 pending_rounds.cc:69] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114381878746161152 op_type: NO_OP noop_request { }
I20250115 02:53:58.466615 13872 raft_consensus.cc:2883] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 02:53:58.467403 13872 raft_consensus.cc:2267] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:58.479997 13602 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b63c3cae1104bd098c4dcea978faa04 with delete_type TABLET_DATA_DELETED (Partition dropped at 2025-01-15 02:53:58 UTC) from {username='slave'} at 127.0.0.1:35548
I20250115 02:53:58.486871 13680 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b63c3cae1104bd098c4dcea978faa04 with delete_type TABLET_DATA_DELETED (Partition dropped at 2025-01-15 02:53:58 UTC) from {username='slave'} at 127.0.0.1:41052
I20250115 02:53:58.501168 13875 tablet_replica.cc:331] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76: stopping tablet replica
I20250115 02:53:58.502110 13875 raft_consensus.cc:2238] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:58.502638 13875 raft_consensus.cc:2267] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:58.505542 13875 ts_tablet_manager.cc:1905] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:58.515676 13874 tablet_replica.cc:331] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325: stopping tablet replica
I20250115 02:53:58.520052 13874 raft_consensus.cc:2238] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:58.520769 13874 raft_consensus.cc:2267] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:58.545651 13874 ts_tablet_manager.cc:1905] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:58.557958 13875 ts_tablet_manager.cc:1918] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 02:53:58.558400 13875 log.cc:1198] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-1-root/wals/5b63c3cae1104bd098c4dcea978faa04
I20250115 02:53:58.517158 13872 ts_tablet_manager.cc:1905] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 02:53:58.568535 13875 ts_tablet_manager.cc:1939] T 5b63c3cae1104bd098c4dcea978faa04 P 14e267b06fb54672bcfbbd9a7ea61b76: Deleting consensus metadata
I20250115 02:53:58.631567 13461 catalog_manager.cc:4872] TS 14e267b06fb54672bcfbbd9a7ea61b76 (127.2.36.66:42909): tablet 5b63c3cae1104bd098c4dcea978faa04 (table other-table [id=c4435e41ca3a4543b823cdcbb898b664]) successfully deleted
I20250115 02:53:58.649086 13874 ts_tablet_manager.cc:1918] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 02:53:58.650974 13874 log.cc:1198] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-0-root/wals/5b63c3cae1104bd098c4dcea978faa04
I20250115 02:53:58.651808 13874 ts_tablet_manager.cc:1939] T 5b63c3cae1104bd098c4dcea978faa04 P bf9b64f185d94f3b9e3bf797f730f325: Deleting consensus metadata
I20250115 02:53:58.669214 13872 ts_tablet_manager.cc:1918] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 02:53:58.670017 13872 log.cc:1198] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84: Deleting WAL directory at /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTableFilters.1736909525896969-2193-0/minicluster-data/ts-2-root/wals/5b63c3cae1104bd098c4dcea978faa04
I20250115 02:53:58.687551 13872 ts_tablet_manager.cc:1939] T 5b63c3cae1104bd098c4dcea978faa04 P 5443d586ce28493ab00dda0282279b84: Deleting consensus metadata
I20250115 02:53:58.691131 13462 catalog_manager.cc:4872] TS 5443d586ce28493ab00dda0282279b84 (127.2.36.67:42515): tablet 5b63c3cae1104bd098c4dcea978faa04 (table other-table [id=c4435e41ca3a4543b823cdcbb898b664]) successfully deleted
I20250115 02:53:58.700695 13461 catalog_manager.cc:4872] TS bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65:41449): tablet 5b63c3cae1104bd098c4dcea978faa04 (table other-table [id=c4435e41ca3a4543b823cdcbb898b664]) successfully deleted
I20250115 02:53:59.289698 13757 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250115 02:53:59.298084 13680 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:53:59.326442 13602 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:53:59.678745 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:53:59.729511 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:59.730131 2193 tablet_replica.cc:331] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325: stopping tablet replica
I20250115 02:53:59.730861 2193 raft_consensus.cc:2238] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:59.731614 2193 raft_consensus.cc:2267] T 96379a1087a24734951ec73f750631fb P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:59.733512 2193 tablet_replica.cc:331] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325: stopping tablet replica
I20250115 02:53:59.734102 2193 raft_consensus.cc:2238] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:59.735375 2193 raft_consensus.cc:2267] T 8a3686c295604093be1507695d8c33e7 P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:59.738269 2193 tablet_replica.cc:331] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325: stopping tablet replica
I20250115 02:53:59.739039 2193 raft_consensus.cc:2238] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:59.739513 2193 raft_consensus.cc:2267] T 9bf81025b11c42baa1536bb073019017 P bf9b64f185d94f3b9e3bf797f730f325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:59.760618 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:53:59.777019 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:53:59.815774 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:59.816485 2193 tablet_replica.cc:331] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76: stopping tablet replica
I20250115 02:53:59.817134 2193 raft_consensus.cc:2238] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:59.817526 2193 raft_consensus.cc:2267] T 9bf81025b11c42baa1536bb073019017 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:59.819393 2193 tablet_replica.cc:331] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76: stopping tablet replica
I20250115 02:53:59.819928 2193 raft_consensus.cc:2238] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:59.820312 2193 raft_consensus.cc:2267] T 8a3686c295604093be1507695d8c33e7 P 14e267b06fb54672bcfbbd9a7ea61b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:59.822469 2193 tablet_replica.cc:331] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76: stopping tablet replica
I20250115 02:53:59.823063 2193 raft_consensus.cc:2238] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:59.823557 2193 raft_consensus.cc:2267] T 96379a1087a24734951ec73f750631fb P 14e267b06fb54672bcfbbd9a7ea61b76 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 02:53:59.841989 13733 consensus_peers.cc:487] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 -> Peer bf9b64f185d94f3b9e3bf797f730f325 (127.2.36.65:41449): Couldn't send request to peer bf9b64f185d94f3b9e3bf797f730f325. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:41449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:53:59.844244 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:53:59.858737 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:53:59.888481 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:53:59.889025 2193 tablet_replica.cc:331] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84: stopping tablet replica
I20250115 02:53:59.889566 2193 raft_consensus.cc:2238] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:59.889969 2193 raft_consensus.cc:2267] T 96379a1087a24734951ec73f750631fb P 5443d586ce28493ab00dda0282279b84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:59.891719 2193 tablet_replica.cc:331] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84: stopping tablet replica
I20250115 02:53:59.892241 2193 raft_consensus.cc:2238] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:59.892980 2193 raft_consensus.cc:2267] T 9bf81025b11c42baa1536bb073019017 P 5443d586ce28493ab00dda0282279b84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:59.894690 2193 tablet_replica.cc:331] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84: stopping tablet replica
I20250115 02:53:59.895148 2193 raft_consensus.cc:2238] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:53:59.895829 2193 raft_consensus.cc:2267] T 8a3686c295604093be1507695d8c33e7 P 5443d586ce28493ab00dda0282279b84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:59.915566 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:53:59.929451 2193 master.cc:537] Master@127.2.36.126:39225 shutting down...
I20250115 02:53:59.944562 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:59.945040 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:59.945287 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P f7269d076a4c4c20ac3ae6117af75e1f: stopping tablet replica
I20250115 02:53:59.964259 2193 master.cc:559] Master@127.2.36.126:39225 shutdown complete.
I20250115 02:53:59.976207 2193 master.cc:537] Master@127.2.36.125:35873 shutting down...
I20250115 02:53:59.992758 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:53:59.993257 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:53:59.993517 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 417108afc1804786b50a49d58251d850: stopping tablet replica
W20250115 02:54:00.004899 13461 consensus_peers.cc:487] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 -> Peer 417108afc1804786b50a49d58251d850 (127.2.36.125:35873): Couldn't send request to peer 417108afc1804786b50a49d58251d850. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.125:35873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:54:00.012275 2193 master.cc:559] Master@127.2.36.125:35873 shutdown complete.
I20250115 02:54:00.025709 2193 master.cc:537] Master@127.2.36.124:41281 shutting down...
I20250115 02:54:00.042961 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:00.043825 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:00.044152 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 984f16f63d3d406a96364e1576789128: stopping tablet replica
I20250115 02:54:00.063796 2193 master.cc:559] Master@127.2.36.124:41281 shutdown complete.
[ OK ] RemoteKsckTest.TestTableFilters (3835 ms)
[ RUN ] RemoteKsckTest.TestTabletFiltersNoMatch
I20250115 02:54:00.093971 2193 test_util.cc:274] Using random seed: -904212226
I20250115 02:54:00.094955 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:41849,127.2.36.125:33107,127.2.36.124:45501
I20250115 02:54:00.096206 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:00.101133 13915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:00.101754 13916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:00.102632 13918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:00.103960 2193 server_base.cc:1034] running on GCE node
I20250115 02:54:00.104763 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:00.104954 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:00.105094 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909640105078 us; error 0 us; skew 500 ppm
I20250115 02:54:00.105619 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:00.107985 2193 webserver.cc:458] Webserver started at http://127.2.36.126:45517/ using document root <none> and password file <none>
I20250115 02:54:00.108446 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:00.108616 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:00.108856 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:00.109902 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044"
format_stamp: "Formatted at 2025-01-15 02:54:00 on dist-test-slave-1lv2"
I20250115 02:54:00.114097 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250115 02:54:00.117300 13923 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:00.118002 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:54:00.118270 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044"
format_stamp: "Formatted at 2025-01-15 02:54:00 on dist-test-slave-1lv2"
I20250115 02:54:00.118542 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:00.153844 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:00.154848 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:00.187697 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:41849
I20250115 02:54:00.187795 13974 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:41849 every 8 connection(s)
I20250115 02:54:00.191234 13975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:00.191423 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:00.196599 13977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:00.197050 13978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:00.199331 13975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:00.199916 2193 server_base.cc:1034] running on GCE node
W20250115 02:54:00.200327 13981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:00.201345 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:00.201682 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:00.201915 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909640201894 us; error 0 us; skew 500 ppm
I20250115 02:54:00.202685 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:00.206188 2193 webserver.cc:458] Webserver started at http://127.2.36.125:38459/ using document root <none> and password file <none>
I20250115 02:54:00.206881 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:00.207122 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:00.207427 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:00.208727 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "da7a07618f1a4e188f11c5415a09f896"
format_stamp: "Formatted at 2025-01-15 02:54:00 on dist-test-slave-1lv2"
I20250115 02:54:00.211681 13975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:00.213850 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
W20250115 02:54:00.216958 13975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:33107: Network error: Client connection negotiation failed: client connection to 127.2.36.125:33107: connect: Connection refused (error 111)
I20250115 02:54:00.218016 13988 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:00.218791 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:54:00.219055 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "da7a07618f1a4e188f11c5415a09f896"
format_stamp: "Formatted at 2025-01-15 02:54:00 on dist-test-slave-1lv2"
I20250115 02:54:00.219314 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:00.227502 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:00.228395 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:00.236424 13975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } attempt: 1
W20250115 02:54:00.241416 13975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:33107: Network error: Client connection negotiation failed: client connection to 127.2.36.125:33107: connect: Connection refused (error 111)
I20250115 02:54:00.263279 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:33107
I20250115 02:54:00.263372 14039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:33107 every 8 connection(s)
I20250115 02:54:00.266949 14040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:00.267057 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:00.272213 14042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:00.273969 14043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:00.275379 14040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:54:00.276271 14045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:00.277148 2193 server_base.cc:1034] running on GCE node
I20250115 02:54:00.278105 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:00.278291 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:00.278448 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909640278433 us; error 0 us; skew 500 ppm
I20250115 02:54:00.278999 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:00.281713 2193 webserver.cc:458] Webserver started at http://127.2.36.124:33077/ using document root <none> and password file <none>
I20250115 02:54:00.282330 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:00.282516 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:00.282792 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:00.283999 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "01a2846b2ec14a6797b5264e437a79eb"
format_stamp: "Formatted at 2025-01-15 02:54:00 on dist-test-slave-1lv2"
I20250115 02:54:00.288594 14040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:00.289635 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250115 02:54:00.293864 14052 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:00.294643 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250115 02:54:00.294963 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "01a2846b2ec14a6797b5264e437a79eb"
format_stamp: "Formatted at 2025-01-15 02:54:00 on dist-test-slave-1lv2"
I20250115 02:54:00.295228 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:00.299381 14040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:54:00.304755 14040 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45501: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45501: connect: Connection refused (error 111)
I20250115 02:54:00.310921 13975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } attempt: 2
I20250115 02:54:00.317366 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:00.318722 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:00.320114 13975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:54:00.324472 13975 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:45501: Network error: Client connection negotiation failed: client connection to 127.2.36.124:45501: connect: Connection refused (error 111)
I20250115 02:54:00.355257 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:45501
I20250115 02:54:00.355338 14104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:45501 every 8 connection(s)
I20250115 02:54:00.358196 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:54:00.358865 14105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:00.361228 14040 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } attempt: 1
I20250115 02:54:00.366108 14105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:00.399381 13975 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } attempt: 1
I20250115 02:54:00.415745 14105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:00.417697 14040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896: Bootstrap starting.
I20250115 02:54:00.425449 14040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:00.425741 13975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044: Bootstrap starting.
I20250115 02:54:00.430568 14105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:00.432092 14040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896: No bootstrap required, opened a new log
I20250115 02:54:00.433367 13975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:00.435773 14040 raft_consensus.cc:357] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } }
I20250115 02:54:00.436915 14040 raft_consensus.cc:383] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:00.437155 14040 raft_consensus.cc:738] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da7a07618f1a4e188f11c5415a09f896, State: Initialized, Role: FOLLOWER
I20250115 02:54:00.437913 14040 consensus_queue.cc:260] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } }
I20250115 02:54:00.439287 13975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044: No bootstrap required, opened a new log
I20250115 02:54:00.442603 13975 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } }
I20250115 02:54:00.443444 13975 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:00.443779 13975 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a18fbe99b7d4facbb8e8bdfd51fa044, State: Initialized, Role: FOLLOWER
I20250115 02:54:00.443789 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } } }
I20250115 02:54:00.444905 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:00.444908 13975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } }
I20250115 02:54:00.445706 14040 sys_catalog.cc:564] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:54:00.447561 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } } }
I20250115 02:54:00.448460 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:00.449368 13975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:54:00.451128 14105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb: Bootstrap starting.
I20250115 02:54:00.457563 14105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:00.463048 14105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb: No bootstrap required, opened a new log
I20250115 02:54:00.466550 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:54:00.465798 14105 raft_consensus.cc:357] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } }
I20250115 02:54:00.467326 14105 raft_consensus.cc:383] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:00.467692 14105 raft_consensus.cc:738] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a2846b2ec14a6797b5264e437a79eb, State: Initialized, Role: FOLLOWER
I20250115 02:54:00.468659 14105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } }
I20250115 02:54:00.469766 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250115 02:54:00.471521 14133 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:54:00.472012 14133 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:54:00.472931 14136 catalog_manager.cc:1559] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:54:00.473207 14136 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:54:00.475466 14137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } } }
I20250115 02:54:00.476032 14137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:00.476784 14105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:54:00.490490 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:00.492843 14148 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:54:00.493217 14148 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:54:00.498524 14149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:00.499186 14150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:00.501674 14152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:00.501813 2193 server_base.cc:1034] running on GCE node
I20250115 02:54:00.502864 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:00.503077 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:00.503225 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909640503211 us; error 0 us; skew 500 ppm
I20250115 02:54:00.503749 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:00.506347 2193 webserver.cc:458] Webserver started at http://127.2.36.65:39495/ using document root <none> and password file <none>
I20250115 02:54:00.506877 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:00.507073 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:00.507344 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:00.508796 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "8e1d0a550d0745a1b31f77a2a52cdb51"
format_stamp: "Formatted at 2025-01-15 02:54:00 on dist-test-slave-1lv2"
I20250115 02:54:00.513525 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250115 02:54:00.517150 14157 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:00.518126 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:54:00.518447 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "8e1d0a550d0745a1b31f77a2a52cdb51"
format_stamp: "Formatted at 2025-01-15 02:54:00 on dist-test-slave-1lv2"
I20250115 02:54:00.518740 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:00.529409 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:00.530632 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:00.532102 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:54:00.534492 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:54:00.534691 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:00.534976 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:54:00.535141 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:00.574970 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:40877
I20250115 02:54:00.575074 14219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:40877 every 8 connection(s)
I20250115 02:54:00.594637 14220 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45501
I20250115 02:54:00.595206 14220 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:00.596184 14221 heartbeater.cc:346] Connected to a master server at 127.2.36.125:33107
I20250115 02:54:00.596220 14220 heartbeater.cc:509] Master 127.2.36.124:45501 requested a full tablet report, sending...
I20250115 02:54:00.596629 14221 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:00.597651 14221 heartbeater.cc:509] Master 127.2.36.125:33107 requested a full tablet report, sending...
I20250115 02:54:00.600154 14070 ts_manager.cc:194] Registered new tserver with Master: 8e1d0a550d0745a1b31f77a2a52cdb51 (127.2.36.65:40877)
I20250115 02:54:00.600697 14005 ts_manager.cc:194] Registered new tserver with Master: 8e1d0a550d0745a1b31f77a2a52cdb51 (127.2.36.65:40877)
I20250115 02:54:00.604101 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:54:00.606213 14222 heartbeater.cc:346] Connected to a master server at 127.2.36.126:41849
I20250115 02:54:00.606657 14222 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:00.607626 14222 heartbeater.cc:509] Master 127.2.36.126:41849 requested a full tablet report, sending...
I20250115 02:54:00.610533 13940 ts_manager.cc:194] Registered new tserver with Master: 8e1d0a550d0745a1b31f77a2a52cdb51 (127.2.36.65:40877)
W20250115 02:54:00.612066 14227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:00.614270 14228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:00.615309 14230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:00.617662 2193 server_base.cc:1034] running on GCE node
I20250115 02:54:00.618510 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:00.618722 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:00.618921 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909640618904 us; error 0 us; skew 500 ppm
I20250115 02:54:00.619463 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:00.621909 2193 webserver.cc:458] Webserver started at http://127.2.36.66:37823/ using document root <none> and password file <none>
I20250115 02:54:00.622354 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:00.622529 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:00.622787 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:00.624261 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "74693c87fe4244cca4d0f0b2a59c1514"
format_stamp: "Formatted at 2025-01-15 02:54:00 on dist-test-slave-1lv2"
I20250115 02:54:00.629858 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250115 02:54:00.633075 14235 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:00.633809 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 02:54:00.634090 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "74693c87fe4244cca4d0f0b2a59c1514"
format_stamp: "Formatted at 2025-01-15 02:54:00 on dist-test-slave-1lv2"
I20250115 02:54:00.634339 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:00.653057 14112 raft_consensus.cc:491] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:00.653635 14112 raft_consensus.cc:513] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } }
I20250115 02:54:00.655748 14112 leader_election.cc:290] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a18fbe99b7d4facbb8e8bdfd51fa044 (127.2.36.126:41849), 01a2846b2ec14a6797b5264e437a79eb (127.2.36.124:45501)
I20250115 02:54:00.657406 13950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da7a07618f1a4e188f11c5415a09f896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" is_pre_election: true
I20250115 02:54:00.657202 14080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da7a07618f1a4e188f11c5415a09f896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a2846b2ec14a6797b5264e437a79eb" is_pre_election: true
I20250115 02:54:00.658190 13950 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da7a07618f1a4e188f11c5415a09f896 in term 0.
I20250115 02:54:00.658298 14080 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da7a07618f1a4e188f11c5415a09f896 in term 0.
I20250115 02:54:00.659597 13992 leader_election.cc:304] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a2846b2ec14a6797b5264e437a79eb, da7a07618f1a4e188f11c5415a09f896; no voters:
I20250115 02:54:00.660452 14112 raft_consensus.cc:2798] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:00.660915 14112 raft_consensus.cc:491] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:00.661312 14112 raft_consensus.cc:3054] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:00.661693 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:00.663363 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:00.669889 14112 raft_consensus.cc:513] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } }
I20250115 02:54:00.670693 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:54:00.671747 14112 leader_election.cc:290] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [CANDIDATE]: Term 1 election: Requested vote from peers 0a18fbe99b7d4facbb8e8bdfd51fa044 (127.2.36.126:41849), 01a2846b2ec14a6797b5264e437a79eb (127.2.36.124:45501)
I20250115 02:54:00.672926 13950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da7a07618f1a4e188f11c5415a09f896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044"
I20250115 02:54:00.673110 14080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da7a07618f1a4e188f11c5415a09f896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a2846b2ec14a6797b5264e437a79eb"
I20250115 02:54:00.673599 13950 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:00.673704 14080 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:00.673970 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:54:00.674325 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:00.674619 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:54:00.674842 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250115 02:54:00.682072 14080 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da7a07618f1a4e188f11c5415a09f896 in term 1.
I20250115 02:54:00.683099 13992 leader_election.cc:304] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a2846b2ec14a6797b5264e437a79eb, da7a07618f1a4e188f11c5415a09f896; no voters:
I20250115 02:54:00.683949 14112 raft_consensus.cc:2798] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:54:00.685006 13950 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da7a07618f1a4e188f11c5415a09f896 in term 1.
I20250115 02:54:00.687626 14112 raft_consensus.cc:695] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 1 LEADER]: Becoming Leader. State: Replica: da7a07618f1a4e188f11c5415a09f896, State: Running, Role: LEADER
I20250115 02:54:00.688844 14112 consensus_queue.cc:237] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } }
I20250115 02:54:00.696497 14244 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da7a07618f1a4e188f11c5415a09f896. Latest consensus state: current_term: 1 leader_uuid: "da7a07618f1a4e188f11c5415a09f896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } } }
I20250115 02:54:00.697232 14244 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [sys.catalog]: This master's current role is: LEADER
I20250115 02:54:00.699206 14254 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:54:00.706902 14254 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:54:00.719245 14080 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [term 1 FOLLOWER]: Refusing update from remote peer da7a07618f1a4e188f11c5415a09f896: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:54:00.721347 13950 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [term 1 FOLLOWER]: Refusing update from remote peer da7a07618f1a4e188f11c5415a09f896: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:54:00.721300 14244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 02:54:00.723166 14244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:00.736132 14137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader da7a07618f1a4e188f11c5415a09f896. Latest consensus state: current_term: 1 leader_uuid: "da7a07618f1a4e188f11c5415a09f896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } } }
I20250115 02:54:00.737008 14137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:00.743288 14254 catalog_manager.cc:1348] Generated new cluster ID: bc9f318988f549879daea775b7c11733
I20250115 02:54:00.743702 14254 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:54:00.744967 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da7a07618f1a4e188f11c5415a09f896. Latest consensus state: current_term: 1 leader_uuid: "da7a07618f1a4e188f11c5415a09f896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } } }
I20250115 02:54:00.745782 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:00.745008 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da7a07618f1a4e188f11c5415a09f896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } } }
I20250115 02:54:00.749437 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [sys.catalog]: This master's current role is: LEADER
I20250115 02:54:00.753108 14137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da7a07618f1a4e188f11c5415a09f896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } } }
I20250115 02:54:00.753918 14137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:00.753608 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da7a07618f1a4e188f11c5415a09f896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } } }
I20250115 02:54:00.754459 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [sys.catalog]: This master's current role is: LEADER
I20250115 02:54:00.757827 14281 mvcc.cc:204] Tried to move back new op lower bound from 7114381888372105216 to 7114381888272089088. Current Snapshot: MvccSnapshot[applied={T|T < 7114381888372105216}]
I20250115 02:54:00.772003 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da7a07618f1a4e188f11c5415a09f896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a18fbe99b7d4facbb8e8bdfd51fa044" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 41849 } } peers { permanent_uuid: "da7a07618f1a4e188f11c5415a09f896" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 33107 } } peers { permanent_uuid: "01a2846b2ec14a6797b5264e437a79eb" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 45501 } } }
I20250115 02:54:00.772809 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:00.799662 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:43597
I20250115 02:54:00.803053 14254 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:54:00.808225 14254 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:54:00.799845 14308 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:43597 every 8 connection(s)
I20250115 02:54:00.829229 14309 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45501
I20250115 02:54:00.829689 14309 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:00.833024 14309 heartbeater.cc:509] Master 127.2.36.124:45501 requested a full tablet report, sending...
I20250115 02:54:00.835772 14070 ts_manager.cc:194] Registered new tserver with Master: 74693c87fe4244cca4d0f0b2a59c1514 (127.2.36.66:43597)
I20250115 02:54:00.849268 14254 catalog_manager.cc:5899] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896: Generated new TSK 0
I20250115 02:54:00.850055 14254 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:54:00.854409 14310 heartbeater.cc:346] Connected to a master server at 127.2.36.125:33107
I20250115 02:54:00.854842 14310 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:00.855528 14310 heartbeater.cc:509] Master 127.2.36.125:33107 requested a full tablet report, sending...
I20250115 02:54:00.857709 14005 ts_manager.cc:194] Registered new tserver with Master: 74693c87fe4244cca4d0f0b2a59c1514 (127.2.36.66:43597)
I20250115 02:54:00.858445 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:54:00.861474 14005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32818
I20250115 02:54:00.864200 14004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32802
I20250115 02:54:00.870285 14313 heartbeater.cc:346] Connected to a master server at 127.2.36.126:41849
I20250115 02:54:00.870882 14313 heartbeater.cc:463] Registering TS with master...
W20250115 02:54:00.872181 14315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:00.872578 14313 heartbeater.cc:509] Master 127.2.36.126:41849 requested a full tablet report, sending...
W20250115 02:54:00.873894 14316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:00.874848 14310 heartbeater.cc:501] Master 127.2.36.125:33107 was elected leader, sending a full tablet report...
I20250115 02:54:00.876107 13940 ts_manager.cc:194] Registered new tserver with Master: 74693c87fe4244cca4d0f0b2a59c1514 (127.2.36.66:43597)
W20250115 02:54:00.876600 14318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:00.878149 2193 server_base.cc:1034] running on GCE node
I20250115 02:54:00.878149 14221 heartbeater.cc:501] Master 127.2.36.125:33107 was elected leader, sending a full tablet report...
I20250115 02:54:00.879177 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:00.879478 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:00.879709 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909640879690 us; error 0 us; skew 500 ppm
I20250115 02:54:00.880441 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:00.884472 2193 webserver.cc:458] Webserver started at http://127.2.36.67:44693/ using document root <none> and password file <none>
I20250115 02:54:00.884954 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:00.885143 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:00.885380 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:00.886493 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "47cf9e58840e48509f0382c5c849c24e"
format_stamp: "Formatted at 2025-01-15 02:54:00 on dist-test-slave-1lv2"
I20250115 02:54:00.891726 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20250115 02:54:00.896920 14323 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:00.897787 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250115 02:54:00.898075 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "47cf9e58840e48509f0382c5c849c24e"
format_stamp: "Formatted at 2025-01-15 02:54:00 on dist-test-slave-1lv2"
I20250115 02:54:00.898319 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFiltersNoMatch.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:00.932777 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:00.934057 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:00.936106 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:54:00.938860 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:54:00.939069 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:00.939281 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:54:00.939440 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:01.023928 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:33477
I20250115 02:54:01.024025 14385 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:33477 every 8 connection(s)
I20250115 02:54:01.047230 14386 heartbeater.cc:346] Connected to a master server at 127.2.36.124:45501
I20250115 02:54:01.047628 14386 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:01.048344 14386 heartbeater.cc:509] Master 127.2.36.124:45501 requested a full tablet report, sending...
I20250115 02:54:01.051082 14070 ts_manager.cc:194] Registered new tserver with Master: 47cf9e58840e48509f0382c5c849c24e (127.2.36.67:33477)
I20250115 02:54:01.053534 14387 heartbeater.cc:346] Connected to a master server at 127.2.36.125:33107
I20250115 02:54:01.053966 14387 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:01.054926 14387 heartbeater.cc:509] Master 127.2.36.125:33107 requested a full tablet report, sending...
I20250115 02:54:01.056931 14004 ts_manager.cc:194] Registered new tserver with Master: 47cf9e58840e48509f0382c5c849c24e (127.2.36.67:33477)
I20250115 02:54:01.059633 14004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32830
I20250115 02:54:01.060781 14388 heartbeater.cc:346] Connected to a master server at 127.2.36.126:41849
I20250115 02:54:01.061208 14388 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:01.061939 14388 heartbeater.cc:509] Master 127.2.36.126:41849 requested a full tablet report, sending...
I20250115 02:54:01.064330 13939 ts_manager.cc:194] Registered new tserver with Master: 47cf9e58840e48509f0382c5c849c24e (127.2.36.67:33477)
I20250115 02:54:01.065014 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.022547621s
I20250115 02:54:01.072886 14387 heartbeater.cc:501] Master 127.2.36.125:33107 was elected leader, sending a full tablet report...
I20250115 02:54:01.117897 14005 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32836:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:54:01.120707 14005 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:54:01.223133 14185 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf97135fdf64b70bd05f061bd4a811a (DEFAULT_TABLE table=ksck-test-table [id=11bfce18a7e94cf4b97a8da98b988070]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:54:01.224610 14185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf97135fdf64b70bd05f061bd4a811a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:01.234584 14266 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf97135fdf64b70bd05f061bd4a811a (DEFAULT_TABLE table=ksck-test-table [id=11bfce18a7e94cf4b97a8da98b988070]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:54:01.236052 14266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf97135fdf64b70bd05f061bd4a811a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:01.239269 14265 tablet_service.cc:1467] Processing CreateTablet for tablet e13db3e8a80d423cba939d7bf63d9f90 (DEFAULT_TABLE table=ksck-test-table [id=11bfce18a7e94cf4b97a8da98b988070]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:54:01.239686 14264 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc040c5a62c4c20b93b6c0b9eb0cdf4 (DEFAULT_TABLE table=ksck-test-table [id=11bfce18a7e94cf4b97a8da98b988070]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:54:01.240808 14265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e13db3e8a80d423cba939d7bf63d9f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:01.241468 14264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dc040c5a62c4c20b93b6c0b9eb0cdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:01.244254 14184 tablet_service.cc:1467] Processing CreateTablet for tablet e13db3e8a80d423cba939d7bf63d9f90 (DEFAULT_TABLE table=ksck-test-table [id=11bfce18a7e94cf4b97a8da98b988070]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:54:01.245589 14184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e13db3e8a80d423cba939d7bf63d9f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:01.248262 14183 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc040c5a62c4c20b93b6c0b9eb0cdf4 (DEFAULT_TABLE table=ksck-test-table [id=11bfce18a7e94cf4b97a8da98b988070]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:54:01.249632 14183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dc040c5a62c4c20b93b6c0b9eb0cdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:01.268604 14351 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf97135fdf64b70bd05f061bd4a811a (DEFAULT_TABLE table=ksck-test-table [id=11bfce18a7e94cf4b97a8da98b988070]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:54:01.270020 14351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf97135fdf64b70bd05f061bd4a811a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:01.279687 14349 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc040c5a62c4c20b93b6c0b9eb0cdf4 (DEFAULT_TABLE table=ksck-test-table [id=11bfce18a7e94cf4b97a8da98b988070]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:54:01.281100 14349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dc040c5a62c4c20b93b6c0b9eb0cdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:01.283905 14350 tablet_service.cc:1467] Processing CreateTablet for tablet e13db3e8a80d423cba939d7bf63d9f90 (DEFAULT_TABLE table=ksck-test-table [id=11bfce18a7e94cf4b97a8da98b988070]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:54:01.285256 14350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e13db3e8a80d423cba939d7bf63d9f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:01.313581 14407 tablet_bootstrap.cc:492] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514: Bootstrap starting.
I20250115 02:54:01.319399 14408 tablet_bootstrap.cc:492] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51: Bootstrap starting.
I20250115 02:54:01.338690 14407 tablet_bootstrap.cc:654] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:01.355064 14408 tablet_bootstrap.cc:654] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:01.358527 14409 tablet_bootstrap.cc:492] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e: Bootstrap starting.
I20250115 02:54:01.362131 14407 tablet_bootstrap.cc:492] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514: No bootstrap required, opened a new log
I20250115 02:54:01.367995 14407 ts_tablet_manager.cc:1397] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.007s
I20250115 02:54:01.372184 14407 raft_consensus.cc:357] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.372948 14407 raft_consensus.cc:383] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:01.373172 14407 raft_consensus.cc:738] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74693c87fe4244cca4d0f0b2a59c1514, State: Initialized, Role: FOLLOWER
I20250115 02:54:01.373879 14407 consensus_queue.cc:260] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.379935 14408 tablet_bootstrap.cc:492] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51: No bootstrap required, opened a new log
I20250115 02:54:01.380582 14408 ts_tablet_manager.cc:1397] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51: Time spent bootstrapping tablet: real 0.062s user 0.009s sys 0.003s
I20250115 02:54:01.383574 14408 raft_consensus.cc:357] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } } peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } }
I20250115 02:54:01.386163 14408 raft_consensus.cc:383] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:01.386543 14408 raft_consensus.cc:738] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1d0a550d0745a1b31f77a2a52cdb51, State: Initialized, Role: FOLLOWER
I20250115 02:54:01.385874 14409 tablet_bootstrap.cc:654] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:01.385612 14407 ts_tablet_manager.cc:1428] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250115 02:54:01.391950 14407 tablet_bootstrap.cc:492] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514: Bootstrap starting.
I20250115 02:54:01.392463 14408 consensus_queue.cc:260] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } } peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } }
I20250115 02:54:01.399140 14408 ts_tablet_manager.cc:1428] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20250115 02:54:01.404731 14407 tablet_bootstrap.cc:654] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:01.406203 14408 tablet_bootstrap.cc:492] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51: Bootstrap starting.
I20250115 02:54:01.407975 14409 tablet_bootstrap.cc:492] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e: No bootstrap required, opened a new log
I20250115 02:54:01.420158 14409 ts_tablet_manager.cc:1397] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.000s
I20250115 02:54:01.423254 14409 raft_consensus.cc:357] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } } peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } }
I20250115 02:54:01.424212 14409 raft_consensus.cc:383] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:01.424582 14409 raft_consensus.cc:738] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cf9e58840e48509f0382c5c849c24e, State: Initialized, Role: FOLLOWER
I20250115 02:54:01.425484 14409 consensus_queue.cc:260] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } } peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } }
I20250115 02:54:01.428728 14408 tablet_bootstrap.cc:654] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:01.442978 14409 ts_tablet_manager.cc:1428] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20250115 02:54:01.445247 14407 tablet_bootstrap.cc:492] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514: No bootstrap required, opened a new log
I20250115 02:54:01.446036 14407 ts_tablet_manager.cc:1397] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514: Time spent bootstrapping tablet: real 0.054s user 0.014s sys 0.009s
I20250115 02:54:01.448900 14408 tablet_bootstrap.cc:492] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51: No bootstrap required, opened a new log
I20250115 02:54:01.449528 14408 ts_tablet_manager.cc:1397] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.001s
I20250115 02:54:01.450500 14407 raft_consensus.cc:357] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.451383 14407 raft_consensus.cc:383] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:01.451704 14407 raft_consensus.cc:738] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74693c87fe4244cca4d0f0b2a59c1514, State: Initialized, Role: FOLLOWER
I20250115 02:54:01.452441 14407 consensus_queue.cc:260] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.454840 14408 raft_consensus.cc:357] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.463208 14408 raft_consensus.cc:383] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:01.461488 14407 ts_tablet_manager.cc:1428] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514: Time spent starting tablet: real 0.015s user 0.003s sys 0.002s
I20250115 02:54:01.459789 14409 tablet_bootstrap.cc:492] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e: Bootstrap starting.
I20250115 02:54:01.465961 14408 raft_consensus.cc:738] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1d0a550d0745a1b31f77a2a52cdb51, State: Initialized, Role: FOLLOWER
I20250115 02:54:01.466921 14408 consensus_queue.cc:260] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.469277 14408 ts_tablet_manager.cc:1428] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250115 02:54:01.470206 14408 tablet_bootstrap.cc:492] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51: Bootstrap starting.
I20250115 02:54:01.474246 14407 tablet_bootstrap.cc:492] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514: Bootstrap starting.
I20250115 02:54:01.476579 14408 tablet_bootstrap.cc:654] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:01.483829 14407 tablet_bootstrap.cc:654] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:01.485100 14133 catalog_manager.cc:1260] Loaded cluster ID: bc9f318988f549879daea775b7c11733
I20250115 02:54:01.484524 14409 tablet_bootstrap.cc:654] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:01.494189 14133 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044: loading cluster ID for follower catalog manager: success
I20250115 02:54:01.503331 14133 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044: acquiring CA information for follower catalog manager: success
I20250115 02:54:01.505638 14409 tablet_bootstrap.cc:492] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e: No bootstrap required, opened a new log
I20250115 02:54:01.490309 14407 tablet_bootstrap.cc:492] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514: No bootstrap required, opened a new log
I20250115 02:54:01.506213 14409 ts_tablet_manager.cc:1397] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.000s
I20250115 02:54:01.506361 14407 ts_tablet_manager.cc:1397] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.000s
I20250115 02:54:01.511020 14409 raft_consensus.cc:357] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.511850 14409 raft_consensus.cc:383] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:01.512279 14409 raft_consensus.cc:738] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cf9e58840e48509f0382c5c849c24e, State: Initialized, Role: FOLLOWER
I20250115 02:54:01.511880 14148 catalog_manager.cc:1260] Loaded cluster ID: bc9f318988f549879daea775b7c11733
I20250115 02:54:01.512974 14407 raft_consensus.cc:357] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } } peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } }
I20250115 02:54:01.516369 14408 tablet_bootstrap.cc:492] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51: No bootstrap required, opened a new log
I20250115 02:54:01.516937 14408 ts_tablet_manager.cc:1397] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.010s
I20250115 02:54:01.513044 14148 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb: loading cluster ID for follower catalog manager: success
I20250115 02:54:01.513054 14409 consensus_queue.cc:260] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.520637 14409 ts_tablet_manager.cc:1428] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250115 02:54:01.524515 14409 tablet_bootstrap.cc:492] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e: Bootstrap starting.
I20250115 02:54:01.515956 14407 raft_consensus.cc:383] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:01.526911 14407 raft_consensus.cc:738] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74693c87fe4244cca4d0f0b2a59c1514, State: Initialized, Role: FOLLOWER
I20250115 02:54:01.515558 14133 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:54:01.533100 14148 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb: acquiring CA information for follower catalog manager: success
I20250115 02:54:01.527894 14407 consensus_queue.cc:260] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } } peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } }
I20250115 02:54:01.535718 14409 tablet_bootstrap.cc:654] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:01.536065 14407 ts_tablet_manager.cc:1428] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20250115 02:54:01.520207 14408 raft_consensus.cc:357] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.536685 14408 raft_consensus.cc:383] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:01.537032 14408 raft_consensus.cc:738] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1d0a550d0745a1b31f77a2a52cdb51, State: Initialized, Role: FOLLOWER
I20250115 02:54:01.541242 14408 consensus_queue.cc:260] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.544178 14408 ts_tablet_manager.cc:1428] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250115 02:54:01.546033 14148 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:54:01.548836 14409 tablet_bootstrap.cc:492] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e: No bootstrap required, opened a new log
I20250115 02:54:01.549322 14409 ts_tablet_manager.cc:1397] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.005s
I20250115 02:54:01.551918 14409 raft_consensus.cc:357] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.552487 14409 raft_consensus.cc:383] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:01.552727 14409 raft_consensus.cc:738] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cf9e58840e48509f0382c5c849c24e, State: Initialized, Role: FOLLOWER
I20250115 02:54:01.553507 14409 consensus_queue.cc:260] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.556221 14409 ts_tablet_manager.cc:1428] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250115 02:54:01.566430 14414 raft_consensus.cc:491] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:01.566987 14414 raft_consensus.cc:513] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } } peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } }
I20250115 02:54:01.569406 14414 leader_election.cc:290] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47cf9e58840e48509f0382c5c849c24e (127.2.36.67:33477), 74693c87fe4244cca4d0f0b2a59c1514 (127.2.36.66:43597)
I20250115 02:54:01.587232 14277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf97135fdf64b70bd05f061bd4a811a" candidate_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74693c87fe4244cca4d0f0b2a59c1514" is_pre_election: true
I20250115 02:54:01.587955 14277 raft_consensus.cc:2463] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1d0a550d0745a1b31f77a2a52cdb51 in term 0.
I20250115 02:54:01.593573 14361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf97135fdf64b70bd05f061bd4a811a" candidate_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47cf9e58840e48509f0382c5c849c24e" is_pre_election: true
I20250115 02:54:01.594362 14361 raft_consensus.cc:2463] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1d0a550d0745a1b31f77a2a52cdb51 in term 0.
I20250115 02:54:01.590648 14159 leader_election.cc:304] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74693c87fe4244cca4d0f0b2a59c1514, 8e1d0a550d0745a1b31f77a2a52cdb51; no voters:
I20250115 02:54:01.599367 14414 raft_consensus.cc:2798] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:01.599807 14414 raft_consensus.cc:491] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:01.600203 14414 raft_consensus.cc:3054] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:01.606022 14414 raft_consensus.cc:513] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } } peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } }
I20250115 02:54:01.615728 14361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf97135fdf64b70bd05f061bd4a811a" candidate_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47cf9e58840e48509f0382c5c849c24e"
I20250115 02:54:01.616144 14414 leader_election.cc:290] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [CANDIDATE]: Term 1 election: Requested vote from peers 47cf9e58840e48509f0382c5c849c24e (127.2.36.67:33477), 74693c87fe4244cca4d0f0b2a59c1514 (127.2.36.66:43597)
I20250115 02:54:01.617094 14361 raft_consensus.cc:3054] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:01.617120 14277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf97135fdf64b70bd05f061bd4a811a" candidate_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74693c87fe4244cca4d0f0b2a59c1514"
I20250115 02:54:01.617909 14277 raft_consensus.cc:3054] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:01.625646 14361 raft_consensus.cc:2463] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1d0a550d0745a1b31f77a2a52cdb51 in term 1.
I20250115 02:54:01.628188 14159 leader_election.cc:304] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47cf9e58840e48509f0382c5c849c24e, 8e1d0a550d0745a1b31f77a2a52cdb51; no voters:
I20250115 02:54:01.627867 14277 raft_consensus.cc:2463] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1d0a550d0745a1b31f77a2a52cdb51 in term 1.
I20250115 02:54:01.631330 14414 raft_consensus.cc:2798] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:54:01.636782 14414 raft_consensus.cc:695] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 LEADER]: Becoming Leader. State: Replica: 8e1d0a550d0745a1b31f77a2a52cdb51, State: Running, Role: LEADER
I20250115 02:54:01.637740 14414 consensus_queue.cc:237] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } } peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } }
I20250115 02:54:01.647434 14412 raft_consensus.cc:491] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:01.647995 14412 raft_consensus.cc:513] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.650460 14412 leader_election.cc:290] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e1d0a550d0745a1b31f77a2a52cdb51 (127.2.36.65:40877), 47cf9e58840e48509f0382c5c849c24e (127.2.36.67:33477)
I20250115 02:54:01.652124 14004 catalog_manager.cc:5526] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e1d0a550d0745a1b31f77a2a52cdb51 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } health_report { overall_health: UNKNOWN } } }
I20250115 02:54:01.668617 14414 raft_consensus.cc:491] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:01.669144 14414 raft_consensus.cc:513] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.671183 14414 leader_election.cc:290] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74693c87fe4244cca4d0f0b2a59c1514 (127.2.36.66:43597), 47cf9e58840e48509f0382c5c849c24e (127.2.36.67:33477)
I20250115 02:54:01.673002 14361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc040c5a62c4c20b93b6c0b9eb0cdf4" candidate_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47cf9e58840e48509f0382c5c849c24e" is_pre_election: true
I20250115 02:54:01.673694 14361 raft_consensus.cc:2463] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1d0a550d0745a1b31f77a2a52cdb51 in term 0.
I20250115 02:54:01.674655 14277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc040c5a62c4c20b93b6c0b9eb0cdf4" candidate_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74693c87fe4244cca4d0f0b2a59c1514" is_pre_election: true
I20250115 02:54:01.674860 14159 leader_election.cc:304] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47cf9e58840e48509f0382c5c849c24e, 8e1d0a550d0745a1b31f77a2a52cdb51; no voters:
I20250115 02:54:01.675374 14277 raft_consensus.cc:2463] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1d0a550d0745a1b31f77a2a52cdb51 in term 0.
I20250115 02:54:01.675710 14414 raft_consensus.cc:2798] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:01.676090 14414 raft_consensus.cc:491] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:01.676414 14414 raft_consensus.cc:3054] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:01.686367 14361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc040c5a62c4c20b93b6c0b9eb0cdf4" candidate_uuid: "74693c87fe4244cca4d0f0b2a59c1514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47cf9e58840e48509f0382c5c849c24e" is_pre_election: true
I20250115 02:54:01.687166 14361 raft_consensus.cc:2463] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74693c87fe4244cca4d0f0b2a59c1514 in term 0.
I20250115 02:54:01.688623 14237 leader_election.cc:304] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47cf9e58840e48509f0382c5c849c24e, 74693c87fe4244cca4d0f0b2a59c1514; no voters:
I20250115 02:54:01.690940 14412 raft_consensus.cc:2798] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:01.691337 14412 raft_consensus.cc:491] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:01.691671 14412 raft_consensus.cc:3054] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:01.687422 14414 raft_consensus.cc:513] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.702311 14412 raft_consensus.cc:513] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.703136 14361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc040c5a62c4c20b93b6c0b9eb0cdf4" candidate_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47cf9e58840e48509f0382c5c849c24e"
I20250115 02:54:01.703773 14361 raft_consensus.cc:3054] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:01.702114 14414 leader_election.cc:290] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [CANDIDATE]: Term 1 election: Requested vote from peers 74693c87fe4244cca4d0f0b2a59c1514 (127.2.36.66:43597), 47cf9e58840e48509f0382c5c849c24e (127.2.36.67:33477)
I20250115 02:54:01.704821 14412 leader_election.cc:290] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [CANDIDATE]: Term 1 election: Requested vote from peers 8e1d0a550d0745a1b31f77a2a52cdb51 (127.2.36.65:40877), 47cf9e58840e48509f0382c5c849c24e (127.2.36.67:33477)
I20250115 02:54:01.711788 14195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc040c5a62c4c20b93b6c0b9eb0cdf4" candidate_uuid: "74693c87fe4244cca4d0f0b2a59c1514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" is_pre_election: true
I20250115 02:54:01.712677 14195 raft_consensus.cc:2388] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74693c87fe4244cca4d0f0b2a59c1514 in current term 1: Already voted for candidate 8e1d0a550d0745a1b31f77a2a52cdb51 in this term.
I20250115 02:54:01.713694 14194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc040c5a62c4c20b93b6c0b9eb0cdf4" candidate_uuid: "74693c87fe4244cca4d0f0b2a59c1514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51"
I20250115 02:54:01.714498 14194 raft_consensus.cc:2388] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 74693c87fe4244cca4d0f0b2a59c1514 in current term 1: Already voted for candidate 8e1d0a550d0745a1b31f77a2a52cdb51 in this term.
I20250115 02:54:01.716897 14277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc040c5a62c4c20b93b6c0b9eb0cdf4" candidate_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74693c87fe4244cca4d0f0b2a59c1514"
I20250115 02:54:01.717047 14360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc040c5a62c4c20b93b6c0b9eb0cdf4" candidate_uuid: "74693c87fe4244cca4d0f0b2a59c1514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47cf9e58840e48509f0382c5c849c24e"
I20250115 02:54:01.717690 14277 raft_consensus.cc:2388] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e1d0a550d0745a1b31f77a2a52cdb51 in current term 1: Already voted for candidate 74693c87fe4244cca4d0f0b2a59c1514 in this term.
I20250115 02:54:01.718295 14237 leader_election.cc:304] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74693c87fe4244cca4d0f0b2a59c1514; no voters: 47cf9e58840e48509f0382c5c849c24e, 8e1d0a550d0745a1b31f77a2a52cdb51
I20250115 02:54:01.721738 14361 raft_consensus.cc:2463] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1d0a550d0745a1b31f77a2a52cdb51 in term 1.
I20250115 02:54:01.723017 14412 raft_consensus.cc:2743] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250115 02:54:01.723806 14159 leader_election.cc:304] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47cf9e58840e48509f0382c5c849c24e, 8e1d0a550d0745a1b31f77a2a52cdb51; no voters: 74693c87fe4244cca4d0f0b2a59c1514
I20250115 02:54:01.724750 14414 raft_consensus.cc:2798] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:54:01.726809 14414 raft_consensus.cc:695] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 LEADER]: Becoming Leader. State: Replica: 8e1d0a550d0745a1b31f77a2a52cdb51, State: Running, Role: LEADER
I20250115 02:54:01.729161 14414 consensus_queue.cc:237] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.741490 14002 catalog_manager.cc:5526] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e1d0a550d0745a1b31f77a2a52cdb51 (127.2.36.65). New cstate: current_term: 1 leader_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } health_report { overall_health: UNKNOWN } } }
I20250115 02:54:01.752101 14416 raft_consensus.cc:491] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:01.752673 14416 raft_consensus.cc:513] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.760251 14416 leader_election.cc:290] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e1d0a550d0745a1b31f77a2a52cdb51 (127.2.36.65:40877), 74693c87fe4244cca4d0f0b2a59c1514 (127.2.36.66:43597)
I20250115 02:54:01.783948 14194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13db3e8a80d423cba939d7bf63d9f90" candidate_uuid: "47cf9e58840e48509f0382c5c849c24e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" is_pre_election: true
I20250115 02:54:01.784345 14277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13db3e8a80d423cba939d7bf63d9f90" candidate_uuid: "47cf9e58840e48509f0382c5c849c24e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74693c87fe4244cca4d0f0b2a59c1514" is_pre_election: true
I20250115 02:54:01.784744 14194 raft_consensus.cc:2463] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47cf9e58840e48509f0382c5c849c24e in term 0.
I20250115 02:54:01.785065 14277 raft_consensus.cc:2463] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47cf9e58840e48509f0382c5c849c24e in term 0.
I20250115 02:54:01.786144 14324 leader_election.cc:304] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47cf9e58840e48509f0382c5c849c24e, 8e1d0a550d0745a1b31f77a2a52cdb51; no voters:
I20250115 02:54:01.786980 14416 raft_consensus.cc:2798] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:01.787348 14416 raft_consensus.cc:491] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:01.787724 14416 raft_consensus.cc:3054] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:01.798823 14416 raft_consensus.cc:513] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.801277 14194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13db3e8a80d423cba939d7bf63d9f90" candidate_uuid: "47cf9e58840e48509f0382c5c849c24e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51"
I20250115 02:54:01.801940 14194 raft_consensus.cc:3054] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:01.802124 14416 leader_election.cc:290] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [CANDIDATE]: Term 1 election: Requested vote from peers 8e1d0a550d0745a1b31f77a2a52cdb51 (127.2.36.65:40877), 74693c87fe4244cca4d0f0b2a59c1514 (127.2.36.66:43597)
I20250115 02:54:01.803182 14277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13db3e8a80d423cba939d7bf63d9f90" candidate_uuid: "47cf9e58840e48509f0382c5c849c24e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74693c87fe4244cca4d0f0b2a59c1514"
I20250115 02:54:01.803723 14277 raft_consensus.cc:3054] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:01.810534 14277 raft_consensus.cc:2463] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47cf9e58840e48509f0382c5c849c24e in term 1.
I20250115 02:54:01.809974 14194 raft_consensus.cc:2463] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47cf9e58840e48509f0382c5c849c24e in term 1.
I20250115 02:54:01.814570 14324 leader_election.cc:304] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47cf9e58840e48509f0382c5c849c24e, 8e1d0a550d0745a1b31f77a2a52cdb51; no voters:
I20250115 02:54:01.815591 14416 raft_consensus.cc:2798] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:54:01.816900 14416 raft_consensus.cc:695] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 1 LEADER]: Becoming Leader. State: Replica: 47cf9e58840e48509f0382c5c849c24e, State: Running, Role: LEADER
I20250115 02:54:01.817719 14416 consensus_queue.cc:237] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } }
I20250115 02:54:01.827178 14002 catalog_manager.cc:5526] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e reported cstate change: term changed from 0 to 1, leader changed from <none> to 47cf9e58840e48509f0382c5c849c24e (127.2.36.67). New cstate: current_term: 1 leader_uuid: "47cf9e58840e48509f0382c5c849c24e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 } health_report { overall_health: HEALTHY } } }
I20250115 02:54:02.075217 14414 consensus_queue.cc:1035] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:02.085620 14414 consensus_queue.cc:1035] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:02.174240 14414 consensus_queue.cc:1035] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47cf9e58840e48509f0382c5c849c24e" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 33477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:02.239118 14414 consensus_queue.cc:1035] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:02.254779 14416 consensus_queue.cc:1035] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1d0a550d0745a1b31f77a2a52cdb51" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 40877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:02.321980 14416 consensus_queue.cc:1035] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [LEADER]: Connected to new peer: Peer: permanent_uuid: "74693c87fe4244cca4d0f0b2a59c1514" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:02.676826 14184 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250115 02:54:02.679355 14351 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:54:02.685142 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:54:03.046166 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:54:03.096755 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:54:03.097360 2193 tablet_replica.cc:331] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51: stopping tablet replica
I20250115 02:54:03.098136 2193 raft_consensus.cc:2238] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:03.098647 2193 raft_consensus.cc:2267] T e13db3e8a80d423cba939d7bf63d9f90 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:03.101430 2193 tablet_replica.cc:331] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51: stopping tablet replica
I20250115 02:54:03.102138 2193 raft_consensus.cc:2238] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:03.103085 2193 raft_consensus.cc:2267] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:03.105679 2193 tablet_replica.cc:331] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51: stopping tablet replica
I20250115 02:54:03.106262 2193 raft_consensus.cc:2238] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:03.107182 2193 raft_consensus.cc:2267] T 9cf97135fdf64b70bd05f061bd4a811a P 8e1d0a550d0745a1b31f77a2a52cdb51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:03.125528 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:54:03.137833 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:54:03.174060 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:54:03.174780 2193 tablet_replica.cc:331] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514: stopping tablet replica
I20250115 02:54:03.175460 2193 raft_consensus.cc:2238] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:03.175920 2193 raft_consensus.cc:2267] T 9cf97135fdf64b70bd05f061bd4a811a P 74693c87fe4244cca4d0f0b2a59c1514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:03.177637 2193 tablet_replica.cc:331] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514: stopping tablet replica
I20250115 02:54:03.178078 2193 raft_consensus.cc:2238] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:03.178438 2193 raft_consensus.cc:2267] T e13db3e8a80d423cba939d7bf63d9f90 P 74693c87fe4244cca4d0f0b2a59c1514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:03.180806 2193 tablet_replica.cc:331] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514: stopping tablet replica
I20250115 02:54:03.181440 2193 raft_consensus.cc:2238] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:03.181890 2193 raft_consensus.cc:2267] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 74693c87fe4244cca4d0f0b2a59c1514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:03.203287 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:54:03.217111 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:54:03.243515 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:54:03.244086 2193 tablet_replica.cc:331] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e: stopping tablet replica
I20250115 02:54:03.244638 2193 raft_consensus.cc:2238] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:03.245378 2193 raft_consensus.cc:2267] T e13db3e8a80d423cba939d7bf63d9f90 P 47cf9e58840e48509f0382c5c849c24e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:03.247377 2193 tablet_replica.cc:331] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e: stopping tablet replica
I20250115 02:54:03.247921 2193 raft_consensus.cc:2238] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:03.248317 2193 raft_consensus.cc:2267] T 3dc040c5a62c4c20b93b6c0b9eb0cdf4 P 47cf9e58840e48509f0382c5c849c24e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:03.250159 2193 tablet_replica.cc:331] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e: stopping tablet replica
I20250115 02:54:03.250789 2193 raft_consensus.cc:2238] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:03.251168 2193 raft_consensus.cc:2267] T 9cf97135fdf64b70bd05f061bd4a811a P 47cf9e58840e48509f0382c5c849c24e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:03.271201 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:54:03.284969 2193 master.cc:537] Master@127.2.36.126:41849 shutting down...
I20250115 02:54:03.301892 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:03.302386 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:03.302736 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 0a18fbe99b7d4facbb8e8bdfd51fa044: stopping tablet replica
I20250115 02:54:03.321581 2193 master.cc:559] Master@127.2.36.126:41849 shutdown complete.
I20250115 02:54:03.335736 2193 master.cc:537] Master@127.2.36.125:33107 shutting down...
I20250115 02:54:03.353559 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:03.354481 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:03.354851 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P da7a07618f1a4e188f11c5415a09f896: stopping tablet replica
I20250115 02:54:03.374647 2193 master.cc:559] Master@127.2.36.125:33107 shutdown complete.
I20250115 02:54:03.388289 2193 master.cc:537] Master@127.2.36.124:45501 shutting down...
I20250115 02:54:03.405081 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:03.405578 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:03.405890 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 01a2846b2ec14a6797b5264e437a79eb: stopping tablet replica
I20250115 02:54:03.424134 2193 master.cc:559] Master@127.2.36.124:45501 shutdown complete.
[ OK ] RemoteKsckTest.TestTabletFiltersNoMatch (3361 ms)
[ RUN ] RemoteKsckTest.TestTabletFilters
I20250115 02:54:03.455390 2193 test_util.cc:274] Using random seed: -900850806
I20250115 02:54:03.456343 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:33307,127.2.36.125:43779,127.2.36.124:43753
I20250115 02:54:03.457594 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:03.462574 14487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:03.462685 14488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:03.463836 14490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:03.465054 2193 server_base.cc:1034] running on GCE node
I20250115 02:54:03.465829 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:03.465999 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:03.466135 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909643466121 us; error 0 us; skew 500 ppm
I20250115 02:54:03.466691 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:03.468978 2193 webserver.cc:458] Webserver started at http://127.2.36.126:34027/ using document root <none> and password file <none>
I20250115 02:54:03.469424 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:03.469589 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:03.469825 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:03.470888 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5"
format_stamp: "Formatted at 2025-01-15 02:54:03 on dist-test-slave-1lv2"
I20250115 02:54:03.475134 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250115 02:54:03.478260 14495 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:03.479002 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:54:03.479274 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5"
format_stamp: "Formatted at 2025-01-15 02:54:03 on dist-test-slave-1lv2"
I20250115 02:54:03.479523 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:03.496299 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:03.497372 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:03.533474 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:33307
I20250115 02:54:03.533571 14546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:33307 every 8 connection(s)
I20250115 02:54:03.537102 14547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:03.537276 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:03.542577 14549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:03.544095 14550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:03.544957 14547 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:03.548085 2193 server_base.cc:1034] running on GCE node
W20250115 02:54:03.548691 14552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:03.550017 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:03.550293 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:03.550489 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909643550471 us; error 0 us; skew 500 ppm
I20250115 02:54:03.551223 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:03.554008 2193 webserver.cc:458] Webserver started at http://127.2.36.125:44797/ using document root <none> and password file <none>
I20250115 02:54:03.554538 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:03.554698 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:03.554991 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:03.556267 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "c035faceb797465bb3da3c516d356a9f"
format_stamp: "Formatted at 2025-01-15 02:54:03 on dist-test-slave-1lv2"
I20250115 02:54:03.558985 14547 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:54:03.561444 14498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.36.125:43779: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20250115 02:54:03.561733 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
W20250115 02:54:03.564338 14547 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:43779: Network error: Client connection negotiation failed: client connection to 127.2.36.125:43779: connect: Connection refused (error 111)
I20250115 02:54:03.565827 14560 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:03.566500 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:54:03.566803 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "c035faceb797465bb3da3c516d356a9f"
format_stamp: "Formatted at 2025-01-15 02:54:03 on dist-test-slave-1lv2"
I20250115 02:54:03.567062 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:03.604626 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:03.605633 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:03.611799 14547 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } attempt: 1
W20250115 02:54:03.616264 14547 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:43779: Network error: Client connection negotiation failed: client connection to 127.2.36.125:43779: connect: Connection refused (error 111)
I20250115 02:54:03.642941 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:43779
I20250115 02:54:03.643038 14611 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:43779 every 8 connection(s)
I20250115 02:54:03.646576 14612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:03.646785 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:03.652249 14614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:03.652665 14615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:03.654251 14612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:54:03.655802 14617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:03.656168 2193 server_base.cc:1034] running on GCE node
I20250115 02:54:03.656775 14547 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } attempt: 2
I20250115 02:54:03.657408 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:03.657675 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:03.657871 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909643657853 us; error 0 us; skew 500 ppm
I20250115 02:54:03.658645 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:03.661759 2193 webserver.cc:458] Webserver started at http://127.2.36.124:40061/ using document root <none> and password file <none>
I20250115 02:54:03.662433 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:03.662678 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:03.663018 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:03.664317 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "3d74faba7f964c9d8facf0bc7d71aa83"
format_stamp: "Formatted at 2025-01-15 02:54:03 on dist-test-slave-1lv2"
I20250115 02:54:03.667214 14612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:03.671274 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250115 02:54:03.672492 14547 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:03.675513 14625 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:03.676515 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250115 02:54:03.676801 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "3d74faba7f964c9d8facf0bc7d71aa83"
format_stamp: "Formatted at 2025-01-15 02:54:03 on dist-test-slave-1lv2"
W20250115 02:54:03.676867 14547 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:43753: Network error: Client connection negotiation failed: client connection to 127.2.36.124:43753: connect: Connection refused (error 111)
I20250115 02:54:03.677209 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:03.686328 14612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:54:03.691431 14612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:43753: Network error: Client connection negotiation failed: client connection to 127.2.36.124:43753: connect: Connection refused (error 111)
I20250115 02:54:03.715965 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:03.717159 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:03.718912 14612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } attempt: 1
W20250115 02:54:03.723203 14612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:43753: Network error: Client connection negotiation failed: client connection to 127.2.36.124:43753: connect: Connection refused (error 111)
I20250115 02:54:03.727459 14547 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } attempt: 1
W20250115 02:54:03.732075 14547 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:43753: Network error: Client connection negotiation failed: client connection to 127.2.36.124:43753: connect: Connection refused (error 111)
I20250115 02:54:03.755707 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:43753
I20250115 02:54:03.755803 14677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:43753 every 8 connection(s)
I20250115 02:54:03.758627 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:54:03.759343 14678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:03.765358 14678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:03.773694 14612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } attempt: 2
I20250115 02:54:03.775852 14678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:03.778520 14547 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } attempt: 2
I20250115 02:54:03.787356 14678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:03.793591 14612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f: Bootstrap starting.
I20250115 02:54:03.799796 14547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5: Bootstrap starting.
I20250115 02:54:03.800151 14612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:03.805325 14678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83: Bootstrap starting.
I20250115 02:54:03.805611 14612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f: No bootstrap required, opened a new log
I20250115 02:54:03.805889 14547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:03.808524 14612 raft_consensus.cc:357] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } }
I20250115 02:54:03.809190 14612 raft_consensus.cc:383] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:03.809463 14612 raft_consensus.cc:738] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c035faceb797465bb3da3c516d356a9f, State: Initialized, Role: FOLLOWER
I20250115 02:54:03.810163 14678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:03.810817 14547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5: No bootstrap required, opened a new log
I20250115 02:54:03.810179 14612 consensus_queue.cc:260] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } }
I20250115 02:54:03.812649 14685 sys_catalog.cc:455] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } } }
I20250115 02:54:03.813607 14685 sys_catalog.cc:458] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:03.813588 14547 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } }
I20250115 02:54:03.814369 14612 sys_catalog.cc:564] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:54:03.814640 14547 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:03.814993 14547 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1963e39eaefb4b3a8fb3a6c2ade09de5, State: Initialized, Role: FOLLOWER
I20250115 02:54:03.815768 14547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } }
I20250115 02:54:03.816351 14678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83: No bootstrap required, opened a new log
I20250115 02:54:03.817814 14687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } } }
I20250115 02:54:03.818604 14687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:03.819373 14678 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } }
I20250115 02:54:03.819748 14547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:54:03.820118 14678 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:03.820499 14678 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d74faba7f964c9d8facf0bc7d71aa83, State: Initialized, Role: FOLLOWER
I20250115 02:54:03.821308 14678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } }
I20250115 02:54:03.823499 14693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } } }
I20250115 02:54:03.824316 14693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:03.825340 14678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [sys.catalog]: configured and running, proceeding with master startup.
W20250115 02:54:03.836969 14705 catalog_manager.cc:1559] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:54:03.837419 14705 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:54:03.839576 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:54:03.839913 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250115 02:54:03.843638 14716 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:54:03.843930 14716 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:54:03.844273 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:03.846556 14721 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:54:03.846927 14721 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:54:03.851565 14722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:03.852161 14723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:03.852979 14725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:03.853283 2193 server_base.cc:1034] running on GCE node
I20250115 02:54:03.854081 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:03.854269 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:03.854411 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909643854388 us; error 0 us; skew 500 ppm
I20250115 02:54:03.854918 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:03.857132 2193 webserver.cc:458] Webserver started at http://127.2.36.65:44449/ using document root <none> and password file <none>
I20250115 02:54:03.857564 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:03.857713 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:03.857911 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:03.859004 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "4293fe47735f4071827e9c6c84e6ac75"
format_stamp: "Formatted at 2025-01-15 02:54:03 on dist-test-slave-1lv2"
I20250115 02:54:03.863128 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250115 02:54:03.866127 14730 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:03.866896 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 02:54:03.867182 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "4293fe47735f4071827e9c6c84e6ac75"
format_stamp: "Formatted at 2025-01-15 02:54:03 on dist-test-slave-1lv2"
I20250115 02:54:03.867449 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:03.897441 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:03.898488 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:03.899853 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:54:03.902051 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:54:03.902246 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:03.902472 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:54:03.902622 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:03.939612 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:32859
I20250115 02:54:03.939713 14792 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:32859 every 8 connection(s)
I20250115 02:54:03.949240 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:54:03.959496 14793 heartbeater.cc:346] Connected to a master server at 127.2.36.124:43753
I20250115 02:54:03.959985 14793 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:03.960698 14795 heartbeater.cc:346] Connected to a master server at 127.2.36.126:33307
I20250115 02:54:03.960891 14793 heartbeater.cc:509] Master 127.2.36.124:43753 requested a full tablet report, sending...
I20250115 02:54:03.961131 14795 heartbeater.cc:463] Registering TS with master...
W20250115 02:54:03.963028 14800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:03.964321 14795 heartbeater.cc:509] Master 127.2.36.126:33307 requested a full tablet report, sending...
I20250115 02:54:03.964943 14643 ts_manager.cc:194] Registered new tserver with Master: 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859)
I20250115 02:54:03.967784 14512 ts_manager.cc:194] Registered new tserver with Master: 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859)
I20250115 02:54:03.972350 14794 heartbeater.cc:346] Connected to a master server at 127.2.36.125:43779
I20250115 02:54:03.972900 14794 heartbeater.cc:463] Registering TS with master...
W20250115 02:54:03.973225 14801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:03.973675 2193 server_base.cc:1034] running on GCE node
I20250115 02:54:03.973860 14794 heartbeater.cc:509] Master 127.2.36.125:43779 requested a full tablet report, sending...
I20250115 02:54:03.972193 14687 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:03.974634 14687 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } }
W20250115 02:54:03.975423 14803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:03.976420 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250115 02:54:03.976583 14687 leader_election.cc:290] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c035faceb797465bb3da3c516d356a9f (127.2.36.125:43779), 3d74faba7f964c9d8facf0bc7d71aa83 (127.2.36.124:43753)
W20250115 02:54:03.976765 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:03.976953 14577 ts_manager.cc:194] Registered new tserver with Master: 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859)
I20250115 02:54:03.977089 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909643977069 us; error 0 us; skew 500 ppm
I20250115 02:54:03.977965 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:03.979451 14653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" is_pre_election: true
I20250115 02:54:03.980204 14653 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1963e39eaefb4b3a8fb3a6c2ade09de5 in term 0.
I20250115 02:54:03.980437 14587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c035faceb797465bb3da3c516d356a9f" is_pre_election: true
I20250115 02:54:03.981168 14587 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1963e39eaefb4b3a8fb3a6c2ade09de5 in term 0.
I20250115 02:54:03.982529 14498 leader_election.cc:304] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1963e39eaefb4b3a8fb3a6c2ade09de5, c035faceb797465bb3da3c516d356a9f; no voters:
I20250115 02:54:03.984330 14687 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:03.984635 14687 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:03.984879 14687 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:03.989723 14687 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } }
I20250115 02:54:03.990820 2193 webserver.cc:458] Webserver started at http://127.2.36.66:36723/ using document root <none> and password file <none>
I20250115 02:54:03.991500 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:03.991746 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:03.991861 14687 leader_election.cc:290] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [CANDIDATE]: Term 1 election: Requested vote from peers c035faceb797465bb3da3c516d356a9f (127.2.36.125:43779), 3d74faba7f964c9d8facf0bc7d71aa83 (127.2.36.124:43753)
I20250115 02:54:03.992055 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:03.992888 14587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c035faceb797465bb3da3c516d356a9f"
I20250115 02:54:03.993286 14653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d74faba7f964c9d8facf0bc7d71aa83"
I20250115 02:54:03.993525 14587 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:03.993566 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "5fef326a741e4a1eb589ee9b4b56054d"
format_stamp: "Formatted at 2025-01-15 02:54:03 on dist-test-slave-1lv2"
I20250115 02:54:03.993939 14653 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:03.999905 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20250115 02:54:04.000176 14587 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1963e39eaefb4b3a8fb3a6c2ade09de5 in term 1.
I20250115 02:54:04.002666 14653 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1963e39eaefb4b3a8fb3a6c2ade09de5 in term 1.
I20250115 02:54:04.004596 14808 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:04.004745 14498 leader_election.cc:304] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1963e39eaefb4b3a8fb3a6c2ade09de5, c035faceb797465bb3da3c516d356a9f; no voters:
I20250115 02:54:04.006031 14687 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:54:04.006379 2193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:54:04.006827 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "5fef326a741e4a1eb589ee9b4b56054d"
format_stamp: "Formatted at 2025-01-15 02:54:03 on dist-test-slave-1lv2"
I20250115 02:54:04.007089 14687 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 1 LEADER]: Becoming Leader. State: Replica: 1963e39eaefb4b3a8fb3a6c2ade09de5, State: Running, Role: LEADER
I20250115 02:54:04.007243 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:04.008060 14687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } }
I20250115 02:54:04.013536 14809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1963e39eaefb4b3a8fb3a6c2ade09de5. Latest consensus state: current_term: 1 leader_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } } }
I20250115 02:54:04.014305 14809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [sys.catalog]: This master's current role is: LEADER
I20250115 02:54:04.016566 14815 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:54:04.022635 14815 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:54:04.030748 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:04.031473 14653 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [term 1 FOLLOWER]: Refusing update from remote peer 1963e39eaefb4b3a8fb3a6c2ade09de5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:54:04.032311 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:04.032620 14587 raft_consensus.cc:1270] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [term 1 FOLLOWER]: Refusing update from remote peer 1963e39eaefb4b3a8fb3a6c2ade09de5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:54:04.034308 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:54:04.035058 14809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:04.035795 14687 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:04.037412 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:54:04.037734 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:04.038025 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:54:04.038229 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:04.051193 14693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1963e39eaefb4b3a8fb3a6c2ade09de5. Latest consensus state: current_term: 1 leader_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } } }
I20250115 02:54:04.052093 14685 sys_catalog.cc:455] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1963e39eaefb4b3a8fb3a6c2ade09de5. Latest consensus state: current_term: 1 leader_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } } }
I20250115 02:54:04.052985 14685 sys_catalog.cc:458] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:04.053342 14693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:04.062920 14815 catalog_manager.cc:1348] Generated new cluster ID: 7e1e09dc9366458bb8e0285c55afad26
I20250115 02:54:04.063360 14815 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:54:04.064405 14687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } } }
I20250115 02:54:04.065327 14687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [sys.catalog]: This master's current role is: LEADER
I20250115 02:54:04.067049 14809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } } }
I20250115 02:54:04.067946 14809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [sys.catalog]: This master's current role is: LEADER
I20250115 02:54:04.078544 14693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } } }
I20250115 02:54:04.079432 14693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:04.080988 14685 sys_catalog.cc:455] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1963e39eaefb4b3a8fb3a6c2ade09de5" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 33307 } } peers { permanent_uuid: "c035faceb797465bb3da3c516d356a9f" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43779 } } peers { permanent_uuid: "3d74faba7f964c9d8facf0bc7d71aa83" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 43753 } } }
I20250115 02:54:04.081773 14685 sys_catalog.cc:458] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:04.122462 14815 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:54:04.124616 14815 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:54:04.146354 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:33679
I20250115 02:54:04.146445 14881 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:33679 every 8 connection(s)
I20250115 02:54:04.169963 14815 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5: Generated new TSK 0
I20250115 02:54:04.170795 14815 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:54:04.181674 14512 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49766
I20250115 02:54:04.184379 14882 heartbeater.cc:346] Connected to a master server at 127.2.36.124:43753
I20250115 02:54:04.184805 14882 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:04.185555 14882 heartbeater.cc:509] Master 127.2.36.124:43753 requested a full tablet report, sending...
I20250115 02:54:04.188177 14643 ts_manager.cc:194] Registered new tserver with Master: 5fef326a741e4a1eb589ee9b4b56054d (127.2.36.66:33679)
I20250115 02:54:04.189571 14883 heartbeater.cc:346] Connected to a master server at 127.2.36.125:43779
I20250115 02:54:04.190075 14883 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:04.191072 14883 heartbeater.cc:509] Master 127.2.36.125:43779 requested a full tablet report, sending...
I20250115 02:54:04.195017 14795 heartbeater.cc:501] Master 127.2.36.126:33307 was elected leader, sending a full tablet report...
I20250115 02:54:04.196197 14577 ts_manager.cc:194] Registered new tserver with Master: 5fef326a741e4a1eb589ee9b4b56054d (127.2.36.66:33679)
I20250115 02:54:04.196545 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:04.206064 14889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:04.208019 14884 heartbeater.cc:346] Connected to a master server at 127.2.36.126:33307
I20250115 02:54:04.208508 14884 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:04.212182 14884 heartbeater.cc:509] Master 127.2.36.126:33307 requested a full tablet report, sending...
I20250115 02:54:04.214646 14512 ts_manager.cc:194] Registered new tserver with Master: 5fef326a741e4a1eb589ee9b4b56054d (127.2.36.66:33679)
I20250115 02:54:04.216866 14512 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49782
W20250115 02:54:04.223745 14890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:04.227041 2193 server_base.cc:1034] running on GCE node
W20250115 02:54:04.227306 14892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:04.228241 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:04.228482 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:04.228663 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909644228650 us; error 0 us; skew 500 ppm
I20250115 02:54:04.229312 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:04.230197 14884 heartbeater.cc:501] Master 127.2.36.126:33307 was elected leader, sending a full tablet report...
I20250115 02:54:04.232842 2193 webserver.cc:458] Webserver started at http://127.2.36.67:33321/ using document root <none> and password file <none>
I20250115 02:54:04.233515 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:04.233762 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:04.234071 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:04.235546 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "10e8b317846646d89a491bfcdd812305"
format_stamp: "Formatted at 2025-01-15 02:54:04 on dist-test-slave-1lv2"
I20250115 02:54:04.240628 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250115 02:54:04.243764 14897 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:04.244693 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:54:04.244985 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "10e8b317846646d89a491bfcdd812305"
format_stamp: "Formatted at 2025-01-15 02:54:04 on dist-test-slave-1lv2"
I20250115 02:54:04.245324 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.TestTabletFilters.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:04.260864 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:04.262166 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:04.264108 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:54:04.267136 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:54:04.267396 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:04.267669 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:54:04.267820 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:04.329207 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:44293
I20250115 02:54:04.329291 14959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:44293 every 8 connection(s)
I20250115 02:54:04.361074 14960 heartbeater.cc:346] Connected to a master server at 127.2.36.124:43753
I20250115 02:54:04.361490 14960 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:04.362293 14960 heartbeater.cc:509] Master 127.2.36.124:43753 requested a full tablet report, sending...
I20250115 02:54:04.366688 14642 ts_manager.cc:194] Registered new tserver with Master: 10e8b317846646d89a491bfcdd812305 (127.2.36.67:44293)
I20250115 02:54:04.367350 14962 heartbeater.cc:346] Connected to a master server at 127.2.36.126:33307
I20250115 02:54:04.367779 14962 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:04.369135 14962 heartbeater.cc:509] Master 127.2.36.126:33307 requested a full tablet report, sending...
I20250115 02:54:04.372963 14961 heartbeater.cc:346] Connected to a master server at 127.2.36.125:43779
I20250115 02:54:04.373366 14961 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:04.374125 14961 heartbeater.cc:509] Master 127.2.36.125:43779 requested a full tablet report, sending...
I20250115 02:54:04.375046 14512 ts_manager.cc:194] Registered new tserver with Master: 10e8b317846646d89a491bfcdd812305 (127.2.36.67:44293)
I20250115 02:54:04.376214 14576 ts_manager.cc:194] Registered new tserver with Master: 10e8b317846646d89a491bfcdd812305 (127.2.36.67:44293)
I20250115 02:54:04.376804 14512 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49786
I20250115 02:54:04.377252 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.025103108s
I20250115 02:54:04.389618 14962 heartbeater.cc:501] Master 127.2.36.126:33307 was elected leader, sending a full tablet report...
I20250115 02:54:04.410683 14511 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49796:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:54:04.413051 14511 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:54:04.530814 14925 tablet_service.cc:1467] Processing CreateTablet for tablet 29397984b9ff4a72bb893194b97249ed (DEFAULT_TABLE table=ksck-test-table [id=0e699e275dcd47ef81a8ef864856525f]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:54:04.531150 14924 tablet_service.cc:1467] Processing CreateTablet for tablet 8d444f88f8104bca857f2db3b10ba566 (DEFAULT_TABLE table=ksck-test-table [id=0e699e275dcd47ef81a8ef864856525f]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:54:04.532224 14925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29397984b9ff4a72bb893194b97249ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:04.533293 14924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d444f88f8104bca857f2db3b10ba566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:04.533905 14923 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba41c984414414cbea6f00b20f2c5de (DEFAULT_TABLE table=ksck-test-table [id=0e699e275dcd47ef81a8ef864856525f]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:54:04.535372 14923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba41c984414414cbea6f00b20f2c5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:04.558359 14979 tablet_bootstrap.cc:492] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305: Bootstrap starting.
I20250115 02:54:04.565045 14847 tablet_service.cc:1467] Processing CreateTablet for tablet 29397984b9ff4a72bb893194b97249ed (DEFAULT_TABLE table=ksck-test-table [id=0e699e275dcd47ef81a8ef864856525f]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:54:04.566326 14846 tablet_service.cc:1467] Processing CreateTablet for tablet 8d444f88f8104bca857f2db3b10ba566 (DEFAULT_TABLE table=ksck-test-table [id=0e699e275dcd47ef81a8ef864856525f]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:54:04.566426 14847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29397984b9ff4a72bb893194b97249ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:04.567725 14846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d444f88f8104bca857f2db3b10ba566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:04.565054 14979 tablet_bootstrap.cc:654] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:04.569561 14845 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba41c984414414cbea6f00b20f2c5de (DEFAULT_TABLE table=ksck-test-table [id=0e699e275dcd47ef81a8ef864856525f]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:54:04.570927 14845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba41c984414414cbea6f00b20f2c5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:04.585166 14981 tablet_bootstrap.cc:492] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d: Bootstrap starting.
I20250115 02:54:04.587489 14979 tablet_bootstrap.cc:492] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305: No bootstrap required, opened a new log
I20250115 02:54:04.588034 14979 ts_tablet_manager.cc:1397] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.004s
I20250115 02:54:04.594810 14981 tablet_bootstrap.cc:654] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:04.599123 14758 tablet_service.cc:1467] Processing CreateTablet for tablet 29397984b9ff4a72bb893194b97249ed (DEFAULT_TABLE table=ksck-test-table [id=0e699e275dcd47ef81a8ef864856525f]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:54:04.600024 14981 tablet_bootstrap.cc:492] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d: No bootstrap required, opened a new log
I20250115 02:54:04.599016 14979 raft_consensus.cc:357] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } }
I20250115 02:54:04.600543 14758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29397984b9ff4a72bb893194b97249ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:04.600555 14981 ts_tablet_manager.cc:1397] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20250115 02:54:04.603740 14981 raft_consensus.cc:357] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } }
I20250115 02:54:04.604480 14981 raft_consensus.cc:383] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:04.604763 14981 raft_consensus.cc:738] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fef326a741e4a1eb589ee9b4b56054d, State: Initialized, Role: FOLLOWER
I20250115 02:54:04.608157 14981 consensus_queue.cc:260] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } }
I20250115 02:54:04.611361 14981 ts_tablet_manager.cc:1428] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d: Time spent starting tablet: real 0.010s user 0.002s sys 0.005s
I20250115 02:54:04.600688 14979 raft_consensus.cc:383] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:04.612037 14979 raft_consensus.cc:738] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e8b317846646d89a491bfcdd812305, State: Initialized, Role: FOLLOWER
I20250115 02:54:04.612426 14981 tablet_bootstrap.cc:492] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d: Bootstrap starting.
I20250115 02:54:04.612919 14979 consensus_queue.cc:260] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } }
I20250115 02:54:04.617924 14757 tablet_service.cc:1467] Processing CreateTablet for tablet 8d444f88f8104bca857f2db3b10ba566 (DEFAULT_TABLE table=ksck-test-table [id=0e699e275dcd47ef81a8ef864856525f]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:54:04.619298 14981 tablet_bootstrap.cc:654] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:04.619632 14756 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba41c984414414cbea6f00b20f2c5de (DEFAULT_TABLE table=ksck-test-table [id=0e699e275dcd47ef81a8ef864856525f]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:54:04.622543 14757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d444f88f8104bca857f2db3b10ba566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:04.636013 14756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba41c984414414cbea6f00b20f2c5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:04.651978 14979 ts_tablet_manager.cc:1428] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305: Time spent starting tablet: real 0.064s user 0.016s sys 0.010s
I20250115 02:54:04.682723 14979 tablet_bootstrap.cc:492] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305: Bootstrap starting.
I20250115 02:54:04.706349 14986 tablet_bootstrap.cc:492] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75: Bootstrap starting.
I20250115 02:54:04.707350 14979 tablet_bootstrap.cc:654] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:04.711102 14981 tablet_bootstrap.cc:492] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d: No bootstrap required, opened a new log
I20250115 02:54:04.711678 14981 ts_tablet_manager.cc:1397] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d: Time spent bootstrapping tablet: real 0.099s user 0.036s sys 0.027s
I20250115 02:54:04.716885 14981 raft_consensus.cc:357] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.717639 14981 raft_consensus.cc:383] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:04.717921 14981 raft_consensus.cc:738] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fef326a741e4a1eb589ee9b4b56054d, State: Initialized, Role: FOLLOWER
I20250115 02:54:04.718683 14981 consensus_queue.cc:260] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.727347 14981 ts_tablet_manager.cc:1428] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250115 02:54:04.730928 14979 tablet_bootstrap.cc:492] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305: No bootstrap required, opened a new log
I20250115 02:54:04.731603 14981 tablet_bootstrap.cc:492] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d: Bootstrap starting.
I20250115 02:54:04.731551 14979 ts_tablet_manager.cc:1397] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.007s
I20250115 02:54:04.734688 14979 raft_consensus.cc:357] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.735574 14979 raft_consensus.cc:383] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:04.735955 14979 raft_consensus.cc:738] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e8b317846646d89a491bfcdd812305, State: Initialized, Role: FOLLOWER
I20250115 02:54:04.736332 14983 raft_consensus.cc:491] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:04.736842 14983 raft_consensus.cc:513] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.739492 14983 leader_election.cc:290] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859), 10e8b317846646d89a491bfcdd812305 (127.2.36.67:44293)
I20250115 02:54:04.736815 14979 consensus_queue.cc:260] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.751673 14981 tablet_bootstrap.cc:654] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:04.751612 14979 ts_tablet_manager.cc:1428] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20250115 02:54:04.764972 14986 tablet_bootstrap.cc:654] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:04.771786 14979 tablet_bootstrap.cc:492] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305: Bootstrap starting.
I20250115 02:54:04.786921 14981 tablet_bootstrap.cc:492] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d: No bootstrap required, opened a new log
I20250115 02:54:04.787477 14981 ts_tablet_manager.cc:1397] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.001s
I20250115 02:54:04.789002 14979 tablet_bootstrap.cc:654] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:04.789836 14768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba41c984414414cbea6f00b20f2c5de" candidate_uuid: "5fef326a741e4a1eb589ee9b4b56054d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4293fe47735f4071827e9c6c84e6ac75" is_pre_election: true
W20250115 02:54:04.791659 14812 leader_election.cc:343] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:54:04.790158 14981 raft_consensus.cc:357] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.795004 14981 raft_consensus.cc:383] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:04.795902 14981 raft_consensus.cc:738] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fef326a741e4a1eb589ee9b4b56054d, State: Initialized, Role: FOLLOWER
I20250115 02:54:04.797866 14981 consensus_queue.cc:260] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.803552 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba41c984414414cbea6f00b20f2c5de" candidate_uuid: "5fef326a741e4a1eb589ee9b4b56054d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e8b317846646d89a491bfcdd812305" is_pre_election: true
W20250115 02:54:04.806052 14812 leader_election.cc:343] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10e8b317846646d89a491bfcdd812305 (127.2.36.67:44293): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:54:04.806509 14812 leader_election.cc:304] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fef326a741e4a1eb589ee9b4b56054d; no voters: 10e8b317846646d89a491bfcdd812305, 4293fe47735f4071827e9c6c84e6ac75
I20250115 02:54:04.807787 14983 raft_consensus.cc:2743] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:54:04.812036 14979 tablet_bootstrap.cc:492] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305: No bootstrap required, opened a new log
I20250115 02:54:04.812647 14979 ts_tablet_manager.cc:1397] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.008s
I20250115 02:54:04.813122 14981 ts_tablet_manager.cc:1428] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d: Time spent starting tablet: real 0.025s user 0.005s sys 0.001s
I20250115 02:54:04.817373 14979 raft_consensus.cc:357] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.818243 14979 raft_consensus.cc:383] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:04.818826 14979 raft_consensus.cc:738] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e8b317846646d89a491bfcdd812305, State: Initialized, Role: FOLLOWER
I20250115 02:54:04.819484 14979 consensus_queue.cc:260] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.821923 14979 ts_tablet_manager.cc:1428] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250115 02:54:04.832194 14986 tablet_bootstrap.cc:492] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75: No bootstrap required, opened a new log
I20250115 02:54:04.835496 14986 ts_tablet_manager.cc:1397] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75: Time spent bootstrapping tablet: real 0.129s user 0.013s sys 0.004s
I20250115 02:54:04.839180 14986 raft_consensus.cc:357] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } }
I20250115 02:54:04.840054 14986 raft_consensus.cc:383] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:04.840430 14986 raft_consensus.cc:738] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4293fe47735f4071827e9c6c84e6ac75, State: Initialized, Role: FOLLOWER
I20250115 02:54:04.841181 14986 consensus_queue.cc:260] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } }
I20250115 02:54:04.844277 14986 ts_tablet_manager.cc:1428] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250115 02:54:04.844455 14705 catalog_manager.cc:1260] Loaded cluster ID: 7e1e09dc9366458bb8e0285c55afad26
I20250115 02:54:04.844763 14705 catalog_manager.cc:1553] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f: loading cluster ID for follower catalog manager: success
I20250115 02:54:04.851779 14705 catalog_manager.cc:1575] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f: acquiring CA information for follower catalog manager: success
I20250115 02:54:04.852779 14986 tablet_bootstrap.cc:492] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75: Bootstrap starting.
I20250115 02:54:04.858600 14705 catalog_manager.cc:1603] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:54:04.861269 14986 tablet_bootstrap.cc:654] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:04.865528 14721 catalog_manager.cc:1260] Loaded cluster ID: 7e1e09dc9366458bb8e0285c55afad26
I20250115 02:54:04.865876 14721 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83: loading cluster ID for follower catalog manager: success
I20250115 02:54:04.878681 14986 tablet_bootstrap.cc:492] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75: No bootstrap required, opened a new log
I20250115 02:54:04.879247 14986 ts_tablet_manager.cc:1397] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.003s
I20250115 02:54:04.880231 14984 raft_consensus.cc:491] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:04.880745 14984 raft_consensus.cc:513] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } }
I20250115 02:54:04.882004 14986 raft_consensus.cc:357] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.882746 14986 raft_consensus.cc:383] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:04.883046 14986 raft_consensus.cc:738] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4293fe47735f4071827e9c6c84e6ac75, State: Initialized, Role: FOLLOWER
I20250115 02:54:04.883838 14986 consensus_queue.cc:260] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.887117 14986 ts_tablet_manager.cc:1428] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250115 02:54:04.890808 14986 tablet_bootstrap.cc:492] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75: Bootstrap starting.
I20250115 02:54:04.893934 14984 leader_election.cc:290] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fef326a741e4a1eb589ee9b4b56054d (127.2.36.66:33679), 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859)
I20250115 02:54:04.896525 14986 tablet_bootstrap.cc:654] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:04.900801 14986 tablet_bootstrap.cc:492] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75: No bootstrap required, opened a new log
I20250115 02:54:04.901201 14986 ts_tablet_manager.cc:1397] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250115 02:54:04.903451 14986 raft_consensus.cc:357] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.904229 14986 raft_consensus.cc:383] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:04.904520 14986 raft_consensus.cc:738] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4293fe47735f4071827e9c6c84e6ac75, State: Initialized, Role: FOLLOWER
I20250115 02:54:04.905261 14986 consensus_queue.cc:260] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.908075 14983 raft_consensus.cc:491] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:04.908486 14983 raft_consensus.cc:513] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } }
I20250115 02:54:04.910234 14983 leader_election.cc:290] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10e8b317846646d89a491bfcdd812305 (127.2.36.67:44293), 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859)
I20250115 02:54:04.911743 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29397984b9ff4a72bb893194b97249ed" candidate_uuid: "5fef326a741e4a1eb589ee9b4b56054d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e8b317846646d89a491bfcdd812305" is_pre_election: true
I20250115 02:54:04.912447 14935 raft_consensus.cc:2463] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fef326a741e4a1eb589ee9b4b56054d in term 0.
I20250115 02:54:04.913461 14768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29397984b9ff4a72bb893194b97249ed" candidate_uuid: "5fef326a741e4a1eb589ee9b4b56054d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4293fe47735f4071827e9c6c84e6ac75" is_pre_election: true
I20250115 02:54:04.914183 14768 raft_consensus.cc:2463] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fef326a741e4a1eb589ee9b4b56054d in term 0.
I20250115 02:54:04.915278 14812 leader_election.cc:304] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4293fe47735f4071827e9c6c84e6ac75, 5fef326a741e4a1eb589ee9b4b56054d; no voters:
I20250115 02:54:04.916252 14983 raft_consensus.cc:2798] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:04.916606 14983 raft_consensus.cc:491] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:04.916921 14983 raft_consensus.cc:3054] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:04.919036 14986 ts_tablet_manager.cc:1428] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75: Time spent starting tablet: real 0.018s user 0.010s sys 0.000s
I20250115 02:54:04.928342 14983 raft_consensus.cc:513] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } }
I20250115 02:54:04.930986 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29397984b9ff4a72bb893194b97249ed" candidate_uuid: "5fef326a741e4a1eb589ee9b4b56054d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e8b317846646d89a491bfcdd812305"
I20250115 02:54:04.931537 14935 raft_consensus.cc:3054] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:04.935317 14721 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83: acquiring CA information for follower catalog manager: success
I20250115 02:54:04.939563 14935 raft_consensus.cc:2463] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fef326a741e4a1eb589ee9b4b56054d in term 1.
I20250115 02:54:04.941908 14721 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:54:04.942401 14768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29397984b9ff4a72bb893194b97249ed" candidate_uuid: "5fef326a741e4a1eb589ee9b4b56054d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4293fe47735f4071827e9c6c84e6ac75"
I20250115 02:54:04.943060 14768 raft_consensus.cc:3054] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:04.941177 14983 leader_election.cc:290] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [CANDIDATE]: Term 1 election: Requested vote from peers 10e8b317846646d89a491bfcdd812305 (127.2.36.67:44293), 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859)
I20250115 02:54:04.954875 14812 leader_election.cc:304] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e8b317846646d89a491bfcdd812305, 5fef326a741e4a1eb589ee9b4b56054d; no voters:
I20250115 02:54:04.956351 14768 raft_consensus.cc:2463] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fef326a741e4a1eb589ee9b4b56054d in term 1.
I20250115 02:54:04.956547 14983 raft_consensus.cc:2798] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:54:04.960247 14984 raft_consensus.cc:491] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:04.960781 14984 raft_consensus.cc:513] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.962699 14984 leader_election.cc:290] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859), 5fef326a741e4a1eb589ee9b4b56054d (127.2.36.66:33679)
I20250115 02:54:04.968288 14983 raft_consensus.cc:695] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 1 LEADER]: Becoming Leader. State: Replica: 5fef326a741e4a1eb589ee9b4b56054d, State: Running, Role: LEADER
I20250115 02:54:04.969307 14983 consensus_queue.cc:237] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } }
I20250115 02:54:04.969864 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba41c984414414cbea6f00b20f2c5de" candidate_uuid: "10e8b317846646d89a491bfcdd812305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fef326a741e4a1eb589ee9b4b56054d" is_pre_election: true
I20250115 02:54:04.970564 14856 raft_consensus.cc:2463] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10e8b317846646d89a491bfcdd812305 in term 0.
I20250115 02:54:04.971633 14857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29397984b9ff4a72bb893194b97249ed" candidate_uuid: "10e8b317846646d89a491bfcdd812305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fef326a741e4a1eb589ee9b4b56054d" is_pre_election: true
I20250115 02:54:04.973068 14899 leader_election.cc:304] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e8b317846646d89a491bfcdd812305, 5fef326a741e4a1eb589ee9b4b56054d; no voters:
I20250115 02:54:04.974244 14984 raft_consensus.cc:2798] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:04.974617 14984 raft_consensus.cc:491] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:04.976603 14984 raft_consensus.cc:3054] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:04.979172 14767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba41c984414414cbea6f00b20f2c5de" candidate_uuid: "10e8b317846646d89a491bfcdd812305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4293fe47735f4071827e9c6c84e6ac75" is_pre_election: true
I20250115 02:54:04.979993 14767 raft_consensus.cc:2463] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10e8b317846646d89a491bfcdd812305 in term 0.
I20250115 02:54:04.979086 14768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29397984b9ff4a72bb893194b97249ed" candidate_uuid: "10e8b317846646d89a491bfcdd812305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4293fe47735f4071827e9c6c84e6ac75" is_pre_election: true
I20250115 02:54:04.981640 14768 raft_consensus.cc:2388] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10e8b317846646d89a491bfcdd812305 in current term 1: Already voted for candidate 5fef326a741e4a1eb589ee9b4b56054d in this term.
I20250115 02:54:04.985153 14899 leader_election.cc:304] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10e8b317846646d89a491bfcdd812305; no voters: 4293fe47735f4071827e9c6c84e6ac75, 5fef326a741e4a1eb589ee9b4b56054d
I20250115 02:54:04.990283 14984 raft_consensus.cc:513] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:04.992259 14984 leader_election.cc:290] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [CANDIDATE]: Term 1 election: Requested vote from peers 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859), 5fef326a741e4a1eb589ee9b4b56054d (127.2.36.66:33679)
I20250115 02:54:04.992770 14984 raft_consensus.cc:2743] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:54:04.988159 14511 catalog_manager.cc:5526] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fef326a741e4a1eb589ee9b4b56054d (127.2.36.66). New cstate: current_term: 1 leader_uuid: "5fef326a741e4a1eb589ee9b4b56054d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } health_report { overall_health: UNKNOWN } } }
I20250115 02:54:05.002017 14768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba41c984414414cbea6f00b20f2c5de" candidate_uuid: "10e8b317846646d89a491bfcdd812305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4293fe47735f4071827e9c6c84e6ac75"
I20250115 02:54:05.002362 14857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba41c984414414cbea6f00b20f2c5de" candidate_uuid: "10e8b317846646d89a491bfcdd812305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fef326a741e4a1eb589ee9b4b56054d"
I20250115 02:54:05.002678 14768 raft_consensus.cc:3054] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:05.003158 14857 raft_consensus.cc:3054] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:05.015939 14768 raft_consensus.cc:2463] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10e8b317846646d89a491bfcdd812305 in term 1.
I20250115 02:54:05.031862 14983 raft_consensus.cc:491] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:05.033892 14983 raft_consensus.cc:513] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:05.035764 14983 leader_election.cc:290] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859), 10e8b317846646d89a491bfcdd812305 (127.2.36.67:44293)
I20250115 02:54:05.037451 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d444f88f8104bca857f2db3b10ba566" candidate_uuid: "5fef326a741e4a1eb589ee9b4b56054d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e8b317846646d89a491bfcdd812305" is_pre_election: true
I20250115 02:54:05.037459 14768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d444f88f8104bca857f2db3b10ba566" candidate_uuid: "5fef326a741e4a1eb589ee9b4b56054d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4293fe47735f4071827e9c6c84e6ac75" is_pre_election: true
I20250115 02:54:05.038259 14768 raft_consensus.cc:2463] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fef326a741e4a1eb589ee9b4b56054d in term 0.
I20250115 02:54:05.039495 14812 leader_election.cc:304] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4293fe47735f4071827e9c6c84e6ac75, 5fef326a741e4a1eb589ee9b4b56054d; no voters:
I20250115 02:54:05.040339 14983 raft_consensus.cc:2798] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:05.040726 14983 raft_consensus.cc:491] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:05.041051 14983 raft_consensus.cc:3054] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:05.032537 14899 leader_election.cc:304] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e8b317846646d89a491bfcdd812305, 4293fe47735f4071827e9c6c84e6ac75; no voters:
I20250115 02:54:05.030577 14857 raft_consensus.cc:2463] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10e8b317846646d89a491bfcdd812305 in term 1.
I20250115 02:54:05.042984 15000 raft_consensus.cc:2798] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:54:05.043808 15000 raft_consensus.cc:695] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 1 LEADER]: Becoming Leader. State: Replica: 10e8b317846646d89a491bfcdd812305, State: Running, Role: LEADER
I20250115 02:54:05.044629 15000 consensus_queue.cc:237] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:05.038198 14935 raft_consensus.cc:2463] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fef326a741e4a1eb589ee9b4b56054d in term 0.
I20250115 02:54:05.057686 14983 raft_consensus.cc:513] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:05.060416 14768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d444f88f8104bca857f2db3b10ba566" candidate_uuid: "5fef326a741e4a1eb589ee9b4b56054d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4293fe47735f4071827e9c6c84e6ac75"
I20250115 02:54:05.061074 14768 raft_consensus.cc:3054] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:05.061225 14983 leader_election.cc:290] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [CANDIDATE]: Term 1 election: Requested vote from peers 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859), 10e8b317846646d89a491bfcdd812305 (127.2.36.67:44293)
I20250115 02:54:05.061424 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d444f88f8104bca857f2db3b10ba566" candidate_uuid: "5fef326a741e4a1eb589ee9b4b56054d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e8b317846646d89a491bfcdd812305"
I20250115 02:54:05.062031 14935 raft_consensus.cc:3054] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:05.068379 14768 raft_consensus.cc:2463] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fef326a741e4a1eb589ee9b4b56054d in term 1.
I20250115 02:54:05.069623 14812 leader_election.cc:304] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4293fe47735f4071827e9c6c84e6ac75, 5fef326a741e4a1eb589ee9b4b56054d; no voters:
I20250115 02:54:05.070108 14935 raft_consensus.cc:2463] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fef326a741e4a1eb589ee9b4b56054d in term 1.
I20250115 02:54:05.070518 14983 raft_consensus.cc:2798] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:54:05.071062 14983 raft_consensus.cc:695] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 1 LEADER]: Becoming Leader. State: Replica: 5fef326a741e4a1eb589ee9b4b56054d, State: Running, Role: LEADER
I20250115 02:54:05.071898 14983 consensus_queue.cc:237] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } }
I20250115 02:54:05.081293 14512 catalog_manager.cc:5526] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10e8b317846646d89a491bfcdd812305 (127.2.36.67). New cstate: current_term: 1 leader_uuid: "10e8b317846646d89a491bfcdd812305" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } health_report { overall_health: HEALTHY } } }
I20250115 02:54:05.088135 14510 catalog_manager.cc:5526] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fef326a741e4a1eb589ee9b4b56054d (127.2.36.66). New cstate: current_term: 1 leader_uuid: "5fef326a741e4a1eb589ee9b4b56054d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 } health_report { overall_health: UNKNOWN } } }
I20250115 02:54:05.416455 14983 consensus_queue.cc:1035] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:05.449347 15000 consensus_queue.cc:1035] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fef326a741e4a1eb589ee9b4b56054d" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 33679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:05.469854 15000 consensus_queue.cc:1035] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:05.470959 14998 consensus_queue.cc:1035] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20250115 02:54:05.554246 14899 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.36.65:32859, user_credentials={real_user=slave}} blocked reactor thread for 45638.6us
I20250115 02:54:05.563433 14998 consensus_queue.cc:1035] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4293fe47735f4071827e9c6c84e6ac75" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 32859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:05.644068 14998 consensus_queue.cc:1035] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10e8b317846646d89a491bfcdd812305" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 44293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:06.036811 14923 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250115 02:54:06.042287 14757 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250115 02:54:06.059036 14846 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250115 02:54:06.417754 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
W20250115 02:54:06.447455 14812 consensus_peers.cc:487] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d -> Peer 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859): Couldn't send request to peer 4293fe47735f4071827e9c6c84e6ac75. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250115 02:54:06.474061 14812 consensus_peers.cc:487] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d -> Peer 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859): Couldn't send request to peer 4293fe47735f4071827e9c6c84e6ac75. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250115 02:54:06.479611 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:54:06.480321 2193 tablet_replica.cc:331] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75: stopping tablet replica
I20250115 02:54:06.481248 2193 raft_consensus.cc:2238] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:06.481829 2193 raft_consensus.cc:2267] T 8d444f88f8104bca857f2db3b10ba566 P 4293fe47735f4071827e9c6c84e6ac75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:06.483950 2193 tablet_replica.cc:331] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75: stopping tablet replica
I20250115 02:54:06.485055 2193 raft_consensus.cc:2238] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:06.485455 2193 raft_consensus.cc:2267] T 0ba41c984414414cbea6f00b20f2c5de P 4293fe47735f4071827e9c6c84e6ac75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:06.490048 2193 tablet_replica.cc:331] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75: stopping tablet replica
I20250115 02:54:06.490700 2193 raft_consensus.cc:2238] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:06.491243 2193 raft_consensus.cc:2267] T 29397984b9ff4a72bb893194b97249ed P 4293fe47735f4071827e9c6c84e6ac75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:06.513535 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:54:06.531164 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:54:06.559834 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:54:06.560781 2193 tablet_replica.cc:331] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d: stopping tablet replica
I20250115 02:54:06.561367 2193 raft_consensus.cc:2238] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:06.562130 2193 raft_consensus.cc:2267] T 8d444f88f8104bca857f2db3b10ba566 P 5fef326a741e4a1eb589ee9b4b56054d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:06.564261 2193 tablet_replica.cc:331] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d: stopping tablet replica
I20250115 02:54:06.564764 2193 raft_consensus.cc:2238] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:06.565146 2193 raft_consensus.cc:2267] T 0ba41c984414414cbea6f00b20f2c5de P 5fef326a741e4a1eb589ee9b4b56054d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:06.566844 2193 tablet_replica.cc:331] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d: stopping tablet replica
I20250115 02:54:06.567381 2193 raft_consensus.cc:2238] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:06.568280 2193 raft_consensus.cc:2267] T 29397984b9ff4a72bb893194b97249ed P 5fef326a741e4a1eb589ee9b4b56054d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:06.589977 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
W20250115 02:54:06.601217 14899 consensus_peers.cc:487] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 -> Peer 4293fe47735f4071827e9c6c84e6ac75 (127.2.36.65:32859): Couldn't send request to peer 4293fe47735f4071827e9c6c84e6ac75. Status: Network error: Client connection negotiation failed: client connection to 127.2.36.65:32859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 02:54:06.605199 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:54:06.633682 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:54:06.634208 2193 tablet_replica.cc:331] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305: stopping tablet replica
I20250115 02:54:06.634728 2193 raft_consensus.cc:2238] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:06.635182 2193 raft_consensus.cc:2267] T 8d444f88f8104bca857f2db3b10ba566 P 10e8b317846646d89a491bfcdd812305 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:06.637238 2193 tablet_replica.cc:331] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305: stopping tablet replica
I20250115 02:54:06.637748 2193 raft_consensus.cc:2238] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:06.638528 2193 raft_consensus.cc:2267] T 0ba41c984414414cbea6f00b20f2c5de P 10e8b317846646d89a491bfcdd812305 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:06.640305 2193 tablet_replica.cc:331] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305: stopping tablet replica
I20250115 02:54:06.640847 2193 raft_consensus.cc:2238] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:06.641211 2193 raft_consensus.cc:2267] T 29397984b9ff4a72bb893194b97249ed P 10e8b317846646d89a491bfcdd812305 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:06.661572 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:54:06.676616 2193 master.cc:537] Master@127.2.36.126:33307 shutting down...
I20250115 02:54:06.691605 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:06.692417 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:06.692745 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 1963e39eaefb4b3a8fb3a6c2ade09de5: stopping tablet replica
I20250115 02:54:06.712157 2193 master.cc:559] Master@127.2.36.126:33307 shutdown complete.
I20250115 02:54:06.726482 2193 master.cc:537] Master@127.2.36.125:43779 shutting down...
I20250115 02:54:06.742745 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:06.743266 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:06.743548 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P c035faceb797465bb3da3c516d356a9f: stopping tablet replica
I20250115 02:54:06.761940 2193 master.cc:559] Master@127.2.36.125:43779 shutdown complete.
I20250115 02:54:06.772897 2193 master.cc:537] Master@127.2.36.124:43753 shutting down...
I20250115 02:54:06.788214 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:06.788691 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:06.788942 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 3d74faba7f964c9d8facf0bc7d71aa83: stopping tablet replica
I20250115 02:54:06.806423 2193 master.cc:559] Master@127.2.36.124:43753 shutdown complete.
[ OK ] RemoteKsckTest.TestTabletFilters (3380 ms)
[ RUN ] RemoteKsckTest.RangeKeysInTabletSummaries
I20250115 02:54:06.835738 2193 test_util.cc:274] Using random seed: -897470458
I20250115 02:54:06.836643 2193 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.36.126:34885,127.2.36.125:43347,127.2.36.124:33697
I20250115 02:54:06.837854 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:06.842536 15070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:06.842725 15071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:06.844049 15073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:06.844954 2193 server_base.cc:1034] running on GCE node
I20250115 02:54:06.845621 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:06.845793 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:06.845906 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909646845895 us; error 0 us; skew 500 ppm
I20250115 02:54:06.846343 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:06.848501 2193 webserver.cc:458] Webserver started at http://127.2.36.126:46697/ using document root <none> and password file <none>
I20250115 02:54:06.848918 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:06.849062 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:06.849256 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:06.850251 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-0-root/instance:
uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c"
format_stamp: "Formatted at 2025-01-15 02:54:06 on dist-test-slave-1lv2"
I20250115 02:54:06.854418 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250115 02:54:06.857417 15078 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:06.858091 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250115 02:54:06.858346 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-0-root
uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c"
format_stamp: "Formatted at 2025-01-15 02:54:06 on dist-test-slave-1lv2"
I20250115 02:54:06.858600 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:06.866963 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:06.867866 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:06.900041 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.126:34885
I20250115 02:54:06.900135 15129 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.126:34885 every 8 connection(s)
I20250115 02:54:06.903574 15130 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:06.903730 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:06.908847 15132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:06.909277 15133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:06.911029 15130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:54:06.912204 15135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:06.913854 2193 server_base.cc:1034] running on GCE node
I20250115 02:54:06.914846 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:06.915031 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:06.915177 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909646915162 us; error 0 us; skew 500 ppm
I20250115 02:54:06.915705 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:06.918133 2193 webserver.cc:458] Webserver started at http://127.2.36.125:41517/ using document root <none> and password file <none>
I20250115 02:54:06.918574 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:06.918746 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:06.919023 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:06.920073 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-1-root/instance:
uuid: "960aa779586e4ea4b7acbfcbaee1cff4"
format_stamp: "Formatted at 2025-01-15 02:54:06 on dist-test-slave-1lv2"
I20250115 02:54:06.922645 15130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:06.925267 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
W20250115 02:54:06.927181 15130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:43347: Network error: Client connection negotiation failed: client connection to 127.2.36.125:43347: connect: Connection refused (error 111)
I20250115 02:54:06.929026 15143 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:06.929752 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250115 02:54:06.930019 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-1-root
uuid: "960aa779586e4ea4b7acbfcbaee1cff4"
format_stamp: "Formatted at 2025-01-15 02:54:06 on dist-test-slave-1lv2"
I20250115 02:54:06.930269 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:06.950590 15130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } attempt: 1
I20250115 02:54:06.952003 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:06.953182 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250115 02:54:06.954475 15130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.125:43347: Network error: Client connection negotiation failed: client connection to 127.2.36.125:43347: connect: Connection refused (error 111)
I20250115 02:54:06.986938 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.125:43347
I20250115 02:54:06.987027 15194 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.125:43347 every 8 connection(s)
I20250115 02:54:06.990414 15195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:06.990654 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:06.995690 15197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:06.996277 15198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:06.997888 15130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } attempt: 2
I20250115 02:54:06.998790 2193 server_base.cc:1034] running on GCE node
W20250115 02:54:06.998975 15200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:07.000878 15195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:07.001045 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:07.001405 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:07.001636 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909647001619 us; error 0 us; skew 500 ppm
I20250115 02:54:07.002394 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:07.007350 2193 webserver.cc:458] Webserver started at http://127.2.36.124:44621/ using document root <none> and password file <none>
I20250115 02:54:07.008018 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:07.008255 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:07.008587 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:07.009860 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-2-root/instance:
uuid: "8ae7c243377948b4b16aca6bfeae4f8c"
format_stamp: "Formatted at 2025-01-15 02:54:07 on dist-test-slave-1lv2"
I20250115 02:54:07.010267 15130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:07.013113 15195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:54:07.015017 15130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:33697: Network error: Client connection negotiation failed: client connection to 127.2.36.124:33697: connect: Connection refused (error 111)
I20250115 02:54:07.015621 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20250115 02:54:07.019882 15207 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:07.020776 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250115 02:54:07.021088 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-2-root
uuid: "8ae7c243377948b4b16aca6bfeae4f8c"
format_stamp: "Formatted at 2025-01-15 02:54:07 on dist-test-slave-1lv2"
I20250115 02:54:07.021437 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:07.023612 15195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } has no permanent_uuid. Determining permanent_uuid...
W20250115 02:54:07.028975 15195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:33697: Network error: Client connection negotiation failed: client connection to 127.2.36.124:33697: connect: Connection refused (error 111)
I20250115 02:54:07.049175 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:07.050266 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:07.051517 15130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } attempt: 1
W20250115 02:54:07.056059 15130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:33697: Network error: Client connection negotiation failed: client connection to 127.2.36.124:33697: connect: Connection refused (error 111)
I20250115 02:54:07.058396 15195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } attempt: 1
W20250115 02:54:07.063066 15195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.36.124:33697: Network error: Client connection negotiation failed: client connection to 127.2.36.124:33697: connect: Connection refused (error 111)
I20250115 02:54:07.087416 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.124:33697
I20250115 02:54:07.087509 15259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.124:33697 every 8 connection(s)
I20250115 02:54:07.090565 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250115 02:54:07.091250 15260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:07.097324 15260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:07.107342 15260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:07.115468 15260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } has no permanent_uuid. Determining permanent_uuid...
I20250115 02:54:07.122510 15130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } attempt: 2
I20250115 02:54:07.131575 15195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } attempt: 2
I20250115 02:54:07.132280 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c: Bootstrap starting.
I20250115 02:54:07.137722 15260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:07.139008 15130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c: Bootstrap starting.
I20250115 02:54:07.142797 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c: No bootstrap required, opened a new log
I20250115 02:54:07.144364 15130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:07.145009 15260 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } }
I20250115 02:54:07.145570 15260 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:07.145805 15260 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae7c243377948b4b16aca6bfeae4f8c, State: Initialized, Role: FOLLOWER
I20250115 02:54:07.146401 15260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } }
I20250115 02:54:07.148348 15267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } } }
I20250115 02:54:07.149085 15267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:07.149515 15130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c: No bootstrap required, opened a new log
I20250115 02:54:07.150094 15260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:54:07.150416 15195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4: Bootstrap starting.
I20250115 02:54:07.152199 15130 raft_consensus.cc:357] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } }
I20250115 02:54:07.152923 15130 raft_consensus.cc:383] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:07.153208 15130 raft_consensus.cc:738] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d0a417fcaa42fd8bf56e436ba3ec5c, State: Initialized, Role: FOLLOWER
I20250115 02:54:07.154111 15130 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } }
I20250115 02:54:07.155354 15195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:07.156280 15271 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } } }
I20250115 02:54:07.157126 15271 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:07.158160 15130 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [sys.catalog]: configured and running, proceeding with master startup.
I20250115 02:54:07.159749 15195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4: No bootstrap required, opened a new log
I20250115 02:54:07.161825 15195 raft_consensus.cc:357] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } }
I20250115 02:54:07.162431 15195 raft_consensus.cc:383] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:07.162707 15195 raft_consensus.cc:738] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 960aa779586e4ea4b7acbfcbaee1cff4, State: Initialized, Role: FOLLOWER
I20250115 02:54:07.163372 15195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } }
I20250115 02:54:07.165069 15277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } } }
I20250115 02:54:07.165869 15277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:07.166900 15195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [sys.catalog]: configured and running, proceeding with master startup.
W20250115 02:54:07.183938 15296 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:54:07.184444 15296 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 02:54:07.184521 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250115 02:54:07.187875 2193 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250115 02:54:07.189659 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 02:54:07.190879 15298 catalog_manager.cc:1559] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:54:07.191179 15298 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:54:07.191918 15303 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 02:54:07.192265 15303 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 02:54:07.195919 15304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:07.196753 15305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:07.198545 2193 server_base.cc:1034] running on GCE node
W20250115 02:54:07.199323 15307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:07.200094 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:07.200294 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:07.200482 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909647200451 us; error 0 us; skew 500 ppm
I20250115 02:54:07.200969 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:07.203186 2193 webserver.cc:458] Webserver started at http://127.2.36.65:46213/ using document root <none> and password file <none>
I20250115 02:54:07.203619 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:07.203785 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:07.204020 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:07.205016 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-0-root/instance:
uuid: "f7cdfedaf2fa43e5a0756932665a381e"
format_stamp: "Formatted at 2025-01-15 02:54:07 on dist-test-slave-1lv2"
I20250115 02:54:07.209183 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20250115 02:54:07.212208 15312 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:07.212939 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250115 02:54:07.213217 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-0-root
uuid: "f7cdfedaf2fa43e5a0756932665a381e"
format_stamp: "Formatted at 2025-01-15 02:54:07 on dist-test-slave-1lv2"
I20250115 02:54:07.213485 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:07.235486 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:07.236449 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:07.237782 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:54:07.239960 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:54:07.240151 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:07.240365 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:54:07.240540 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:07.276724 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.65:38457
I20250115 02:54:07.276829 15374 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.65:38457 every 8 connection(s)
I20250115 02:54:07.292032 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:54:07.296207 15375 heartbeater.cc:346] Connected to a master server at 127.2.36.124:33697
I20250115 02:54:07.296648 15375 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:07.297480 15375 heartbeater.cc:509] Master 127.2.36.124:33697 requested a full tablet report, sending...
I20250115 02:54:07.298424 15376 heartbeater.cc:346] Connected to a master server at 127.2.36.125:43347
I20250115 02:54:07.299090 15376 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:07.300055 15376 heartbeater.cc:509] Master 127.2.36.125:43347 requested a full tablet report, sending...
I20250115 02:54:07.300408 15225 ts_manager.cc:194] Registered new tserver with Master: f7cdfedaf2fa43e5a0756932665a381e (127.2.36.65:38457)
I20250115 02:54:07.303016 15160 ts_manager.cc:194] Registered new tserver with Master: f7cdfedaf2fa43e5a0756932665a381e (127.2.36.65:38457)
I20250115 02:54:07.305127 2193 server_base.cc:1034] running on GCE node
W20250115 02:54:07.306244 15382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:07.306780 15385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:07.307238 15383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:07.308238 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:07.308434 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:07.308614 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909647308595 us; error 0 us; skew 500 ppm
I20250115 02:54:07.309242 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:07.310328 15377 heartbeater.cc:346] Connected to a master server at 127.2.36.126:34885
I20250115 02:54:07.310741 15377 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:07.311565 15377 heartbeater.cc:509] Master 127.2.36.126:34885 requested a full tablet report, sending...
I20250115 02:54:07.311966 2193 webserver.cc:458] Webserver started at http://127.2.36.66:43237/ using document root <none> and password file <none>
I20250115 02:54:07.312554 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:07.312709 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:07.312899 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:07.313999 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-1-root/instance:
uuid: "1c3ad48800f54a30951c7a1e8baf02bb"
format_stamp: "Formatted at 2025-01-15 02:54:07 on dist-test-slave-1lv2"
I20250115 02:54:07.314277 15095 ts_manager.cc:194] Registered new tserver with Master: f7cdfedaf2fa43e5a0756932665a381e (127.2.36.65:38457)
I20250115 02:54:07.319504 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20250115 02:54:07.322723 15390 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:07.323480 2193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 02:54:07.323786 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-1-root
uuid: "1c3ad48800f54a30951c7a1e8baf02bb"
format_stamp: "Formatted at 2025-01-15 02:54:07 on dist-test-slave-1lv2"
I20250115 02:54:07.324023 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:07.345955 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:07.346916 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:07.348179 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:54:07.350265 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:54:07.350442 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:07.350659 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:54:07.350841 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250115 02:54:07.394145 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.66:43155
I20250115 02:54:07.394237 15452 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.66:43155 every 8 connection(s)
I20250115 02:54:07.412758 15453 heartbeater.cc:346] Connected to a master server at 127.2.36.124:33697
I20250115 02:54:07.413230 15453 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:07.414057 15453 heartbeater.cc:509] Master 127.2.36.124:33697 requested a full tablet report, sending...
I20250115 02:54:07.422066 15225 ts_manager.cc:194] Registered new tserver with Master: 1c3ad48800f54a30951c7a1e8baf02bb (127.2.36.66:43155)
I20250115 02:54:07.425058 2193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 02:54:07.425637 15454 heartbeater.cc:346] Connected to a master server at 127.2.36.125:43347
I20250115 02:54:07.425805 15455 heartbeater.cc:346] Connected to a master server at 127.2.36.126:34885
I20250115 02:54:07.426046 15454 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:07.426258 15455 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:07.426878 15454 heartbeater.cc:509] Master 127.2.36.125:43347 requested a full tablet report, sending...
I20250115 02:54:07.427124 15455 heartbeater.cc:509] Master 127.2.36.126:34885 requested a full tablet report, sending...
I20250115 02:54:07.429145 15160 ts_manager.cc:194] Registered new tserver with Master: 1c3ad48800f54a30951c7a1e8baf02bb (127.2.36.66:43155)
I20250115 02:54:07.429704 15095 ts_manager.cc:194] Registered new tserver with Master: 1c3ad48800f54a30951c7a1e8baf02bb (127.2.36.66:43155)
W20250115 02:54:07.438246 15460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:07.439587 15461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 02:54:07.440883 15463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 02:54:07.441073 2193 server_base.cc:1034] running on GCE node
I20250115 02:54:07.442106 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250115 02:54:07.442320 2193 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250115 02:54:07.442493 2193 hybrid_clock.cc:648] HybridClock initialized: now 1736909647442476 us; error 0 us; skew 500 ppm
I20250115 02:54:07.443163 2193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 02:54:07.446338 2193 webserver.cc:458] Webserver started at http://127.2.36.67:33053/ using document root <none> and password file <none>
I20250115 02:54:07.446993 2193 fs_manager.cc:362] Metadata directory not provided
I20250115 02:54:07.447222 2193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 02:54:07.447494 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 02:54:07.448845 2193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-2-root/instance:
uuid: "b3c8272666bc45a29a8233aa9a9fe9fc"
format_stamp: "Formatted at 2025-01-15 02:54:07 on dist-test-slave-1lv2"
I20250115 02:54:07.453189 2193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250115 02:54:07.457149 15468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:07.458189 2193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250115 02:54:07.458487 2193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-2-root
uuid: "b3c8272666bc45a29a8233aa9a9fe9fc"
format_stamp: "Formatted at 2025-01-15 02:54:07 on dist-test-slave-1lv2"
I20250115 02:54:07.458743 2193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskB0ERDW/test-tmp/ksck_remote-test.0.RemoteKsckTest.RangeKeysInTabletSummaries.1736909525896969-2193-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 02:54:07.478920 15271 raft_consensus.cc:491] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:07.479460 15271 raft_consensus.cc:513] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } }
I20250115 02:54:07.481323 15271 leader_election.cc:290] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 960aa779586e4ea4b7acbfcbaee1cff4 (127.2.36.125:43347), 8ae7c243377948b4b16aca6bfeae4f8c (127.2.36.124:33697)
I20250115 02:54:07.482000 15170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" is_pre_election: true
I20250115 02:54:07.482810 15170 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3d0a417fcaa42fd8bf56e436ba3ec5c in term 0.
I20250115 02:54:07.482828 15235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" is_pre_election: true
I20250115 02:54:07.484113 15080 leader_election.cc:304] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 960aa779586e4ea4b7acbfcbaee1cff4, e3d0a417fcaa42fd8bf56e436ba3ec5c; no voters:
I20250115 02:54:07.484701 2193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 02:54:07.484833 15271 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:07.485314 15271 raft_consensus.cc:491] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:07.485631 15271 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:07.484984 15235 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3d0a417fcaa42fd8bf56e436ba3ec5c in term 0.
I20250115 02:54:07.486107 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 02:54:07.489681 2193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 02:54:07.492820 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 02:54:07.493018 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:07.493244 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 02:54:07.493397 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 02:54:07.495769 15271 raft_consensus.cc:513] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } }
I20250115 02:54:07.498332 15170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "960aa779586e4ea4b7acbfcbaee1cff4"
I20250115 02:54:07.498872 15170 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:07.505581 15170 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3d0a417fcaa42fd8bf56e436ba3ec5c in term 1.
I20250115 02:54:07.507556 15080 leader_election.cc:304] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 960aa779586e4ea4b7acbfcbaee1cff4, e3d0a417fcaa42fd8bf56e436ba3ec5c; no voters:
I20250115 02:54:07.508299 15271 leader_election.cc:290] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [CANDIDATE]: Term 1 election: Requested vote from peers 960aa779586e4ea4b7acbfcbaee1cff4 (127.2.36.125:43347), 8ae7c243377948b4b16aca6bfeae4f8c (127.2.36.124:33697)
I20250115 02:54:07.508929 15271 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:54:07.509358 15271 raft_consensus.cc:695] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 1 LEADER]: Becoming Leader. State: Replica: e3d0a417fcaa42fd8bf56e436ba3ec5c, State: Running, Role: LEADER
I20250115 02:54:07.510174 15271 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } }
I20250115 02:54:07.513051 15235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae7c243377948b4b16aca6bfeae4f8c"
I20250115 02:54:07.513772 15235 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:07.528050 15235 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3d0a417fcaa42fd8bf56e436ba3ec5c in term 1.
I20250115 02:54:07.527829 15480 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3d0a417fcaa42fd8bf56e436ba3ec5c. Latest consensus state: current_term: 1 leader_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } } }
I20250115 02:54:07.528869 15480 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [sys.catalog]: This master's current role is: LEADER
I20250115 02:54:07.531740 15484 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 02:54:07.540100 15484 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 02:54:07.561309 15235 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [term 1 FOLLOWER]: Refusing update from remote peer e3d0a417fcaa42fd8bf56e436ba3ec5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:54:07.563968 15170 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [term 1 FOLLOWER]: Refusing update from remote peer e3d0a417fcaa42fd8bf56e436ba3ec5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 02:54:07.564216 15480 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:07.565510 15271 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:07.581295 15277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3d0a417fcaa42fd8bf56e436ba3ec5c. Latest consensus state: current_term: 1 leader_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } } }
I20250115 02:54:07.582126 15277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:07.597477 15267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3d0a417fcaa42fd8bf56e436ba3ec5c. Latest consensus state: current_term: 1 leader_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } } }
I20250115 02:54:07.598341 15267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:07.609325 15267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } } }
I20250115 02:54:07.610229 15267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:07.609856 15277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } } }
I20250115 02:54:07.612915 15277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 02:54:07.617934 15480 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } } }
I20250115 02:54:07.618713 15480 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [sys.catalog]: This master's current role is: LEADER
I20250115 02:54:07.619239 15480 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d0a417fcaa42fd8bf56e436ba3ec5c" member_type: VOTER last_known_addr { host: "127.2.36.126" port: 34885 } } peers { permanent_uuid: "960aa779586e4ea4b7acbfcbaee1cff4" member_type: VOTER last_known_addr { host: "127.2.36.125" port: 43347 } } peers { permanent_uuid: "8ae7c243377948b4b16aca6bfeae4f8c" member_type: VOTER last_known_addr { host: "127.2.36.124" port: 33697 } } }
I20250115 02:54:07.619915 15480 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [sys.catalog]: This master's current role is: LEADER
I20250115 02:54:07.620746 15484 catalog_manager.cc:1348] Generated new cluster ID: 509edcc5c547481a8289699084a4ee12
I20250115 02:54:07.621081 15484 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 02:54:07.656719 2193 rpc_server.cc:307] RPC server started. Bound to: 127.2.36.67:36949
I20250115 02:54:07.656781 15542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.36.67:36949 every 8 connection(s)
I20250115 02:54:07.696637 15543 heartbeater.cc:346] Connected to a master server at 127.2.36.124:33697
I20250115 02:54:07.697182 15543 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:07.698160 15543 heartbeater.cc:509] Master 127.2.36.124:33697 requested a full tablet report, sending...
I20250115 02:54:07.711354 15224 ts_manager.cc:194] Registered new tserver with Master: b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949)
I20250115 02:54:07.728634 15484 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 02:54:07.730307 15484 catalog_manager.cc:1505] Loading token signing keys...
I20250115 02:54:07.730430 15544 heartbeater.cc:346] Connected to a master server at 127.2.36.125:43347
I20250115 02:54:07.730960 15544 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:07.731776 15544 heartbeater.cc:509] Master 127.2.36.125:43347 requested a full tablet report, sending...
I20250115 02:54:07.745690 15160 ts_manager.cc:194] Registered new tserver with Master: b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949)
I20250115 02:54:07.762620 15546 heartbeater.cc:346] Connected to a master server at 127.2.36.126:34885
I20250115 02:54:07.763224 15546 heartbeater.cc:463] Registering TS with master...
I20250115 02:54:07.764143 15546 heartbeater.cc:509] Master 127.2.36.126:34885 requested a full tablet report, sending...
I20250115 02:54:07.765296 15484 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c: Generated new TSK 0
I20250115 02:54:07.765982 15484 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 02:54:07.769390 15094 ts_manager.cc:194] Registered new tserver with Master: b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949)
I20250115 02:54:07.770241 2193 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007516265s
I20250115 02:54:07.780750 15094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42414
I20250115 02:54:07.784543 15095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42424
I20250115 02:54:07.790447 15095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42416
I20250115 02:54:07.793987 15377 heartbeater.cc:501] Master 127.2.36.126:34885 was elected leader, sending a full tablet report...
I20250115 02:54:07.797618 15546 heartbeater.cc:501] Master 127.2.36.126:34885 was elected leader, sending a full tablet report...
I20250115 02:54:07.805672 15455 heartbeater.cc:501] Master 127.2.36.126:34885 was elected leader, sending a full tablet report...
I20250115 02:54:07.826474 15095 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42434:
name: "ksck-test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\n\000\000\000\004\001\024\000\000\000""\004\001\n\000\000\000\004\001\024\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 02:54:07.831055 15095 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table ksck-test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 02:54:07.923822 15418 tablet_service.cc:1467] Processing CreateTablet for tablet e8515ec1292c45b383bb3c159b7d421b (DEFAULT_TABLE table=ksck-test-table [id=aa97228f1583407ebed585b7517f5043]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:54:07.924793 15417 tablet_service.cc:1467] Processing CreateTablet for tablet a8e4d4320eb14ab8a760b05d98fa71dd (DEFAULT_TABLE table=ksck-test-table [id=aa97228f1583407ebed585b7517f5043]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:54:07.925251 15418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8515ec1292c45b383bb3c159b7d421b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:07.926122 15416 tablet_service.cc:1467] Processing CreateTablet for tablet 85bf22a601334d00bc6af0848cd41a20 (DEFAULT_TABLE table=ksck-test-table [id=aa97228f1583407ebed585b7517f5043]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:54:07.927186 15416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85bf22a601334d00bc6af0848cd41a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:07.928697 15417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e4d4320eb14ab8a760b05d98fa71dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:07.964769 15339 tablet_service.cc:1467] Processing CreateTablet for tablet a8e4d4320eb14ab8a760b05d98fa71dd (DEFAULT_TABLE table=ksck-test-table [id=aa97228f1583407ebed585b7517f5043]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:54:07.965111 15340 tablet_service.cc:1467] Processing CreateTablet for tablet e8515ec1292c45b383bb3c159b7d421b (DEFAULT_TABLE table=ksck-test-table [id=aa97228f1583407ebed585b7517f5043]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:54:07.966231 15339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e4d4320eb14ab8a760b05d98fa71dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:07.967792 15340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8515ec1292c45b383bb3c159b7d421b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:07.968644 15338 tablet_service.cc:1467] Processing CreateTablet for tablet 85bf22a601334d00bc6af0848cd41a20 (DEFAULT_TABLE table=ksck-test-table [id=aa97228f1583407ebed585b7517f5043]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:54:07.969903 15338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85bf22a601334d00bc6af0848cd41a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:08.023689 15563 tablet_bootstrap.cc:492] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e: Bootstrap starting.
I20250115 02:54:08.031256 15563 tablet_bootstrap.cc:654] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:08.044297 15564 tablet_bootstrap.cc:492] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb: Bootstrap starting.
I20250115 02:54:08.048251 15563 tablet_bootstrap.cc:492] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e: No bootstrap required, opened a new log
I20250115 02:54:08.048780 15563 ts_tablet_manager.cc:1397] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.007s
I20250115 02:54:08.052671 15564 tablet_bootstrap.cc:654] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:08.056136 15563 raft_consensus.cc:357] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.056969 15563 raft_consensus.cc:383] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:08.057263 15563 raft_consensus.cc:738] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cdfedaf2fa43e5a0756932665a381e, State: Initialized, Role: FOLLOWER
I20250115 02:54:08.058023 15563 consensus_queue.cc:260] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.076671 15499 tablet_service.cc:1467] Processing CreateTablet for tablet e8515ec1292c45b383bb3c159b7d421b (DEFAULT_TABLE table=ksck-test-table [id=aa97228f1583407ebed585b7517f5043]), partition=RANGE (key) PARTITION VALUES < 10
I20250115 02:54:08.078208 15499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8515ec1292c45b383bb3c159b7d421b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:08.078560 15564 tablet_bootstrap.cc:492] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb: No bootstrap required, opened a new log
I20250115 02:54:08.079150 15564 ts_tablet_manager.cc:1397] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.022s
I20250115 02:54:08.079655 15498 tablet_service.cc:1467] Processing CreateTablet for tablet a8e4d4320eb14ab8a760b05d98fa71dd (DEFAULT_TABLE table=ksck-test-table [id=aa97228f1583407ebed585b7517f5043]), partition=RANGE (key) PARTITION 10 <= VALUES < 20
I20250115 02:54:08.080938 15498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e4d4320eb14ab8a760b05d98fa71dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:08.081920 15564 raft_consensus.cc:357] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.082640 15564 raft_consensus.cc:383] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:08.082952 15564 raft_consensus.cc:738] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3ad48800f54a30951c7a1e8baf02bb, State: Initialized, Role: FOLLOWER
I20250115 02:54:08.083590 15564 consensus_queue.cc:260] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.083830 15497 tablet_service.cc:1467] Processing CreateTablet for tablet 85bf22a601334d00bc6af0848cd41a20 (DEFAULT_TABLE table=ksck-test-table [id=aa97228f1583407ebed585b7517f5043]), partition=RANGE (key) PARTITION 20 <= VALUES
I20250115 02:54:08.085232 15497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85bf22a601334d00bc6af0848cd41a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 02:54:08.128469 15568 raft_consensus.cc:491] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:08.129627 15568 raft_consensus.cc:513] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.132575 15568 leader_election.cc:290] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3ad48800f54a30951c7a1e8baf02bb (127.2.36.66:43155), b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949)
I20250115 02:54:08.136384 15564 ts_tablet_manager.cc:1428] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb: Time spent starting tablet: real 0.057s user 0.016s sys 0.034s
I20250115 02:54:08.139140 15564 tablet_bootstrap.cc:492] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb: Bootstrap starting.
I20250115 02:54:08.142251 15563 ts_tablet_manager.cc:1428] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e: Time spent starting tablet: real 0.092s user 0.008s sys 0.028s
I20250115 02:54:08.146791 15563 tablet_bootstrap.cc:492] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e: Bootstrap starting.
I20250115 02:54:08.147344 15564 tablet_bootstrap.cc:654] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:08.157006 15563 tablet_bootstrap.cc:654] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:08.163578 15428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8515ec1292c45b383bb3c159b7d421b" candidate_uuid: "f7cdfedaf2fa43e5a0756932665a381e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" is_pre_election: true
W20250115 02:54:08.164786 15313 leader_election.cc:343] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c3ad48800f54a30951c7a1e8baf02bb (127.2.36.66:43155): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:54:08.165935 15564 tablet_bootstrap.cc:492] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb: No bootstrap required, opened a new log
I20250115 02:54:08.166420 15564 ts_tablet_manager.cc:1397] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.000s
I20250115 02:54:08.170424 15564 raft_consensus.cc:357] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.171147 15564 raft_consensus.cc:383] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:08.171535 15564 raft_consensus.cc:738] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3ad48800f54a30951c7a1e8baf02bb, State: Initialized, Role: FOLLOWER
I20250115 02:54:08.172279 15564 consensus_queue.cc:260] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.179173 15564 ts_tablet_manager.cc:1428] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb: Time spent starting tablet: real 0.012s user 0.007s sys 0.001s
I20250115 02:54:08.184401 15564 tablet_bootstrap.cc:492] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb: Bootstrap starting.
I20250115 02:54:08.190105 15296 catalog_manager.cc:1260] Loaded cluster ID: 509edcc5c547481a8289699084a4ee12
I20250115 02:54:08.190448 15296 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c: loading cluster ID for follower catalog manager: success
I20250115 02:54:08.197630 15303 catalog_manager.cc:1260] Loaded cluster ID: 509edcc5c547481a8289699084a4ee12
I20250115 02:54:08.197938 15303 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4: loading cluster ID for follower catalog manager: success
I20250115 02:54:08.201625 15564 tablet_bootstrap.cc:654] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:08.204666 15303 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4: acquiring CA information for follower catalog manager: success
I20250115 02:54:08.206780 15296 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c: acquiring CA information for follower catalog manager: success
I20250115 02:54:08.211896 15296 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:54:08.214313 15303 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 02:54:08.300498 15564 tablet_bootstrap.cc:492] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb: No bootstrap required, opened a new log
I20250115 02:54:08.301080 15564 ts_tablet_manager.cc:1397] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb: Time spent bootstrapping tablet: real 0.117s user 0.019s sys 0.022s
I20250115 02:54:08.303790 15564 raft_consensus.cc:357] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.304441 15564 raft_consensus.cc:383] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:08.304663 15564 raft_consensus.cc:738] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3ad48800f54a30951c7a1e8baf02bb, State: Initialized, Role: FOLLOWER
I20250115 02:54:08.305253 15564 consensus_queue.cc:260] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.307289 15564 ts_tablet_manager.cc:1428] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250115 02:54:08.329798 15563 tablet_bootstrap.cc:492] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e: No bootstrap required, opened a new log
I20250115 02:54:08.331211 15563 ts_tablet_manager.cc:1397] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e: Time spent bootstrapping tablet: real 0.185s user 0.020s sys 0.056s
W20250115 02:54:08.331527 15314 leader_election.cc:343] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949): Not found: Tablet not found: e8515ec1292c45b383bb3c159b7d421b
I20250115 02:54:08.333586 15314 leader_election.cc:304] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7cdfedaf2fa43e5a0756932665a381e; no voters: 1c3ad48800f54a30951c7a1e8baf02bb, b3c8272666bc45a29a8233aa9a9fe9fc
I20250115 02:54:08.335247 15568 raft_consensus.cc:2743] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:54:08.339361 15576 tablet_bootstrap.cc:492] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc: Bootstrap starting.
I20250115 02:54:08.344024 15563 raft_consensus.cc:357] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.344907 15563 raft_consensus.cc:383] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:08.345258 15563 raft_consensus.cc:738] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cdfedaf2fa43e5a0756932665a381e, State: Initialized, Role: FOLLOWER
I20250115 02:54:08.346011 15563 consensus_queue.cc:260] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.362958 15563 ts_tablet_manager.cc:1428] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
I20250115 02:54:08.365643 15563 tablet_bootstrap.cc:492] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e: Bootstrap starting.
I20250115 02:54:08.372001 15569 raft_consensus.cc:491] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:08.372574 15569 raft_consensus.cc:513] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.373350 15576 tablet_bootstrap.cc:654] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:08.375053 15569 leader_election.cc:290] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7cdfedaf2fa43e5a0756932665a381e (127.2.36.65:38457), b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949)
I20250115 02:54:08.394649 15563 tablet_bootstrap.cc:654] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:08.405164 15576 tablet_bootstrap.cc:492] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc: No bootstrap required, opened a new log
I20250115 02:54:08.405725 15576 ts_tablet_manager.cc:1397] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.008s
I20250115 02:54:08.408416 15576 raft_consensus.cc:357] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.414001 15576 raft_consensus.cc:383] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:08.414333 15576 raft_consensus.cc:738] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c8272666bc45a29a8233aa9a9fe9fc, State: Initialized, Role: FOLLOWER
I20250115 02:54:08.415171 15576 consensus_queue.cc:260] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.431813 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bf22a601334d00bc6af0848cd41a20" candidate_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" is_pre_election: true
W20250115 02:54:08.433450 15392 leader_election.cc:343] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:54:08.433864 15350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bf22a601334d00bc6af0848cd41a20" candidate_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cdfedaf2fa43e5a0756932665a381e" is_pre_election: true
I20250115 02:54:08.434487 15350 raft_consensus.cc:2463] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3ad48800f54a30951c7a1e8baf02bb in term 0.
I20250115 02:54:08.436918 15576 ts_tablet_manager.cc:1428] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc: Time spent starting tablet: real 0.031s user 0.008s sys 0.000s
I20250115 02:54:08.437752 15391 leader_election.cc:304] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c3ad48800f54a30951c7a1e8baf02bb, f7cdfedaf2fa43e5a0756932665a381e; no voters: b3c8272666bc45a29a8233aa9a9fe9fc
I20250115 02:54:08.438087 15576 tablet_bootstrap.cc:492] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc: Bootstrap starting.
I20250115 02:54:08.438939 15569 raft_consensus.cc:2798] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:08.440403 15569 raft_consensus.cc:491] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:08.440804 15569 raft_consensus.cc:3054] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:08.444533 15576 tablet_bootstrap.cc:654] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:08.448017 15563 tablet_bootstrap.cc:492] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e: No bootstrap required, opened a new log
I20250115 02:54:08.448511 15563 ts_tablet_manager.cc:1397] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e: Time spent bootstrapping tablet: real 0.083s user 0.025s sys 0.010s
I20250115 02:54:08.451165 15563 raft_consensus.cc:357] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.452039 15563 raft_consensus.cc:383] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:08.452330 15563 raft_consensus.cc:738] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cdfedaf2fa43e5a0756932665a381e, State: Initialized, Role: FOLLOWER
I20250115 02:54:08.453114 15563 consensus_queue.cc:260] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.464126 15563 ts_tablet_manager.cc:1428] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250115 02:54:08.465713 15583 raft_consensus.cc:491] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:08.466315 15583 raft_consensus.cc:513] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.469125 15350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e4d4320eb14ab8a760b05d98fa71dd" candidate_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cdfedaf2fa43e5a0756932665a381e" is_pre_election: true
I20250115 02:54:08.469805 15350 raft_consensus.cc:2463] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3ad48800f54a30951c7a1e8baf02bb in term 0.
I20250115 02:54:08.471050 15391 leader_election.cc:304] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3ad48800f54a30951c7a1e8baf02bb, f7cdfedaf2fa43e5a0756932665a381e; no voters:
I20250115 02:54:08.472364 15569 raft_consensus.cc:513] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.472931 15576 tablet_bootstrap.cc:492] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc: No bootstrap required, opened a new log
I20250115 02:54:08.473773 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e4d4320eb14ab8a760b05d98fa71dd" candidate_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" is_pre_election: true
I20250115 02:54:08.478706 15576 ts_tablet_manager.cc:1397] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.018s
I20250115 02:54:08.479010 15518 raft_consensus.cc:2463] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3ad48800f54a30951c7a1e8baf02bb in term 0.
I20250115 02:54:08.481643 15576 raft_consensus.cc:357] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.482476 15576 raft_consensus.cc:383] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:08.482859 15576 raft_consensus.cc:738] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c8272666bc45a29a8233aa9a9fe9fc, State: Initialized, Role: FOLLOWER
I20250115 02:54:08.483318 15350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bf22a601334d00bc6af0848cd41a20" candidate_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cdfedaf2fa43e5a0756932665a381e"
I20250115 02:54:08.483961 15350 raft_consensus.cc:3054] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:08.483582 15576 consensus_queue.cc:260] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.485944 15576 ts_tablet_manager.cc:1428] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250115 02:54:08.488397 15576 tablet_bootstrap.cc:492] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc: Bootstrap starting.
I20250115 02:54:08.472553 15584 raft_consensus.cc:2798] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:08.476601 15583 leader_election.cc:290] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7cdfedaf2fa43e5a0756932665a381e (127.2.36.65:38457), b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949)
I20250115 02:54:08.494078 15576 tablet_bootstrap.cc:654] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc: Neither blocks nor log segments found. Creating new log.
I20250115 02:54:08.495193 15350 raft_consensus.cc:2463] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3ad48800f54a30951c7a1e8baf02bb in term 1.
I20250115 02:54:08.496408 15391 leader_election.cc:304] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3ad48800f54a30951c7a1e8baf02bb, f7cdfedaf2fa43e5a0756932665a381e; no voters:
I20250115 02:54:08.497372 15583 raft_consensus.cc:2798] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:54:08.504725 15584 raft_consensus.cc:491] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:08.505637 15584 raft_consensus.cc:3054] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:08.509604 15569 leader_election.cc:290] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 election: Requested vote from peers f7cdfedaf2fa43e5a0756932665a381e (127.2.36.65:38457), b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949)
I20250115 02:54:08.510699 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bf22a601334d00bc6af0848cd41a20" candidate_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc"
W20250115 02:54:08.513172 15392 leader_election.cc:343] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949): Illegal state: must be running to vote when last-logged opid is not known
I20250115 02:54:08.513888 15583 raft_consensus.cc:695] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 LEADER]: Becoming Leader. State: Replica: 1c3ad48800f54a30951c7a1e8baf02bb, State: Running, Role: LEADER
I20250115 02:54:08.519582 15584 raft_consensus.cc:513] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.514899 15583 consensus_queue.cc:237] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.527336 15576 tablet_bootstrap.cc:492] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc: No bootstrap required, opened a new log
I20250115 02:54:08.527705 15584 leader_election.cc:290] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 election: Requested vote from peers f7cdfedaf2fa43e5a0756932665a381e (127.2.36.65:38457), b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949)
I20250115 02:54:08.527928 15576 ts_tablet_manager.cc:1397] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.009s
I20250115 02:54:08.528017 15350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e4d4320eb14ab8a760b05d98fa71dd" candidate_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cdfedaf2fa43e5a0756932665a381e"
I20250115 02:54:08.528738 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e4d4320eb14ab8a760b05d98fa71dd" candidate_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc"
I20250115 02:54:08.528761 15350 raft_consensus.cc:3054] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:08.529392 15518 raft_consensus.cc:3054] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:08.530921 15576 raft_consensus.cc:357] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.531688 15576 raft_consensus.cc:383] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 02:54:08.532009 15576 raft_consensus.cc:738] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c8272666bc45a29a8233aa9a9fe9fc, State: Initialized, Role: FOLLOWER
I20250115 02:54:08.532811 15576 consensus_queue.cc:260] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.536430 15576 ts_tablet_manager.cc:1428] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250115 02:54:08.538869 15350 raft_consensus.cc:2463] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3ad48800f54a30951c7a1e8baf02bb in term 1.
I20250115 02:54:08.540047 15391 leader_election.cc:304] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3ad48800f54a30951c7a1e8baf02bb, f7cdfedaf2fa43e5a0756932665a381e; no voters:
I20250115 02:54:08.544885 15094 catalog_manager.cc:5526] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c3ad48800f54a30951c7a1e8baf02bb (127.2.36.66). New cstate: current_term: 1 leader_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250115 02:54:08.548792 15583 raft_consensus.cc:2798] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:54:08.549569 15518 raft_consensus.cc:2463] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3ad48800f54a30951c7a1e8baf02bb in term 1.
I20250115 02:54:08.549804 15583 raft_consensus.cc:695] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 LEADER]: Becoming Leader. State: Replica: 1c3ad48800f54a30951c7a1e8baf02bb, State: Running, Role: LEADER
I20250115 02:54:08.550900 15583 consensus_queue.cc:237] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.591894 15094 catalog_manager.cc:5526] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c3ad48800f54a30951c7a1e8baf02bb (127.2.36.66). New cstate: current_term: 1 leader_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250115 02:54:08.682108 15583 raft_consensus.cc:491] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:08.682644 15583 raft_consensus.cc:513] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.684628 15583 leader_election.cc:290] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7cdfedaf2fa43e5a0756932665a381e (127.2.36.65:38457), b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949)
I20250115 02:54:08.685288 15350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8515ec1292c45b383bb3c159b7d421b" candidate_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cdfedaf2fa43e5a0756932665a381e" is_pre_election: true
I20250115 02:54:08.685520 15581 raft_consensus.cc:491] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:08.686053 15350 raft_consensus.cc:2463] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3ad48800f54a30951c7a1e8baf02bb in term 0.
I20250115 02:54:08.686360 15581 raft_consensus.cc:513] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.686803 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8515ec1292c45b383bb3c159b7d421b" candidate_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" is_pre_election: true
I20250115 02:54:08.687234 15391 leader_election.cc:304] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3ad48800f54a30951c7a1e8baf02bb, f7cdfedaf2fa43e5a0756932665a381e; no voters:
I20250115 02:54:08.687489 15518 raft_consensus.cc:2463] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3ad48800f54a30951c7a1e8baf02bb in term 0.
I20250115 02:54:08.688872 15581 leader_election.cc:290] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3ad48800f54a30951c7a1e8baf02bb (127.2.36.66:43155), f7cdfedaf2fa43e5a0756932665a381e (127.2.36.65:38457)
I20250115 02:54:08.688957 15583 raft_consensus.cc:2798] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 02:54:08.691231 15583 raft_consensus.cc:491] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 02:54:08.691573 15583 raft_consensus.cc:3054] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:08.701367 15583 raft_consensus.cc:513] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.704164 15350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8515ec1292c45b383bb3c159b7d421b" candidate_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cdfedaf2fa43e5a0756932665a381e"
I20250115 02:54:08.704617 15583 leader_election.cc:290] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 election: Requested vote from peers f7cdfedaf2fa43e5a0756932665a381e (127.2.36.65:38457), b3c8272666bc45a29a8233aa9a9fe9fc (127.2.36.67:36949)
I20250115 02:54:08.705178 15350 raft_consensus.cc:3054] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:08.710255 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8515ec1292c45b383bb3c159b7d421b" candidate_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc"
I20250115 02:54:08.711002 15518 raft_consensus.cc:3054] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:08.721101 15350 raft_consensus.cc:2463] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3ad48800f54a30951c7a1e8baf02bb in term 1.
I20250115 02:54:08.722369 15391 leader_election.cc:304] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3ad48800f54a30951c7a1e8baf02bb, f7cdfedaf2fa43e5a0756932665a381e; no voters:
I20250115 02:54:08.723251 15583 raft_consensus.cc:2798] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 FOLLOWER]: Leader election won for term 1
I20250115 02:54:08.723711 15583 raft_consensus.cc:695] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 LEADER]: Becoming Leader. State: Replica: 1c3ad48800f54a30951c7a1e8baf02bb, State: Running, Role: LEADER
I20250115 02:54:08.724314 15583 consensus_queue.cc:237] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.725508 15518 raft_consensus.cc:2463] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3ad48800f54a30951c7a1e8baf02bb in term 1.
I20250115 02:54:08.727198 15350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8515ec1292c45b383bb3c159b7d421b" candidate_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cdfedaf2fa43e5a0756932665a381e" is_pre_election: true
I20250115 02:54:08.727805 15350 raft_consensus.cc:2388] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3c8272666bc45a29a8233aa9a9fe9fc in current term 1: Already voted for candidate 1c3ad48800f54a30951c7a1e8baf02bb in this term.
I20250115 02:54:08.730928 15581 raft_consensus.cc:491] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 02:54:08.731451 15581 raft_consensus.cc:513] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } } peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } }
I20250115 02:54:08.735329 15581 leader_election.cc:290] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7cdfedaf2fa43e5a0756932665a381e (127.2.36.65:38457), 1c3ad48800f54a30951c7a1e8baf02bb (127.2.36.66:43155)
I20250115 02:54:08.738152 15350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bf22a601334d00bc6af0848cd41a20" candidate_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7cdfedaf2fa43e5a0756932665a381e" is_pre_election: true
I20250115 02:54:08.739027 15350 raft_consensus.cc:2388] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3c8272666bc45a29a8233aa9a9fe9fc in current term 1: Already voted for candidate 1c3ad48800f54a30951c7a1e8baf02bb in this term.
I20250115 02:54:08.748291 15092 catalog_manager.cc:5526] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c3ad48800f54a30951c7a1e8baf02bb (127.2.36.66). New cstate: current_term: 1 leader_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" member_type: VOTER last_known_addr { host: "127.2.36.66" port: 43155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250115 02:54:08.754479 15427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bf22a601334d00bc6af0848cd41a20" candidate_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" is_pre_election: true
I20250115 02:54:08.759778 15469 leader_election.cc:304] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3c8272666bc45a29a8233aa9a9fe9fc; no voters: 1c3ad48800f54a30951c7a1e8baf02bb, f7cdfedaf2fa43e5a0756932665a381e
I20250115 02:54:08.760972 15581 raft_consensus.cc:3054] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc [term 0 FOLLOWER]: Advancing to term 1
I20250115 02:54:08.754547 15428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8515ec1292c45b383bb3c159b7d421b" candidate_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3ad48800f54a30951c7a1e8baf02bb" is_pre_election: true
I20250115 02:54:08.767216 15469 leader_election.cc:304] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3c8272666bc45a29a8233aa9a9fe9fc; no voters: 1c3ad48800f54a30951c7a1e8baf02bb, f7cdfedaf2fa43e5a0756932665a381e
I20250115 02:54:08.771229 15581 raft_consensus.cc:2743] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:54:08.772205 15581 raft_consensus.cc:2743] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 02:54:08.909060 15583 consensus_queue.cc:1035] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:08.934896 15584 consensus_queue.cc:1035] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:08.942461 15584 consensus_queue.cc:1035] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7cdfedaf2fa43e5a0756932665a381e" member_type: VOTER last_known_addr { host: "127.2.36.65" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 02:54:08.966933 15583 consensus_queue.cc:1035] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3c8272666bc45a29a8233aa9a9fe9fc" member_type: VOTER last_known_addr { host: "127.2.36.67" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/ksck_remote-test.cc:836: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20250115 02:54:08.981264 2193 tablet_server.cc:178] TabletServer@127.2.36.65:0 shutting down...
I20250115 02:54:09.048609 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:54:09.049371 2193 tablet_replica.cc:331] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e: stopping tablet replica
I20250115 02:54:09.050021 2193 raft_consensus.cc:2238] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:09.050457 2193 raft_consensus.cc:2267] T a8e4d4320eb14ab8a760b05d98fa71dd P f7cdfedaf2fa43e5a0756932665a381e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:09.055038 2193 tablet_replica.cc:331] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e: stopping tablet replica
I20250115 02:54:09.055807 2193 raft_consensus.cc:2238] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:09.056380 2193 raft_consensus.cc:2267] T 85bf22a601334d00bc6af0848cd41a20 P f7cdfedaf2fa43e5a0756932665a381e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:09.059027 2193 tablet_replica.cc:331] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e: stopping tablet replica
I20250115 02:54:09.059516 2193 raft_consensus.cc:2238] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:09.059991 2193 raft_consensus.cc:2267] T e8515ec1292c45b383bb3c159b7d421b P f7cdfedaf2fa43e5a0756932665a381e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:09.076521 2193 tablet_server.cc:195] TabletServer@127.2.36.65:0 shutdown complete.
I20250115 02:54:09.092365 2193 tablet_server.cc:178] TabletServer@127.2.36.66:0 shutting down...
I20250115 02:54:09.121848 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:54:09.122376 2193 tablet_replica.cc:331] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb: stopping tablet replica
I20250115 02:54:09.122941 2193 raft_consensus.cc:2238] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:09.123731 2193 raft_consensus.cc:2267] T a8e4d4320eb14ab8a760b05d98fa71dd P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:09.125778 2193 tablet_replica.cc:331] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb: stopping tablet replica
I20250115 02:54:09.126302 2193 raft_consensus.cc:2238] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:09.126864 2193 pending_rounds.cc:62] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb: Trying to abort 1 pending ops.
I20250115 02:54:09.127018 2193 pending_rounds.cc:69] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114381921182490624 op_type: NO_OP noop_request { }
I20250115 02:54:09.127360 2193 raft_consensus.cc:2883] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 02:54:09.127642 2193 raft_consensus.cc:2267] T e8515ec1292c45b383bb3c159b7d421b P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:09.129722 2193 tablet_replica.cc:331] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb: stopping tablet replica
I20250115 02:54:09.130189 2193 raft_consensus.cc:2238] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:09.131044 2193 raft_consensus.cc:2267] T 85bf22a601334d00bc6af0848cd41a20 P 1c3ad48800f54a30951c7a1e8baf02bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:09.146055 2193 tablet_server.cc:195] TabletServer@127.2.36.66:0 shutdown complete.
I20250115 02:54:09.156996 2193 tablet_server.cc:178] TabletServer@127.2.36.67:0 shutting down...
I20250115 02:54:09.180560 2193 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250115 02:54:09.181120 2193 tablet_replica.cc:331] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc: stopping tablet replica
I20250115 02:54:09.181672 2193 raft_consensus.cc:2238] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:09.182101 2193 raft_consensus.cc:2267] T 85bf22a601334d00bc6af0848cd41a20 P b3c8272666bc45a29a8233aa9a9fe9fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:09.184317 2193 tablet_replica.cc:331] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc: stopping tablet replica
I20250115 02:54:09.184827 2193 raft_consensus.cc:2238] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:09.185179 2193 raft_consensus.cc:2267] T e8515ec1292c45b383bb3c159b7d421b P b3c8272666bc45a29a8233aa9a9fe9fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:09.186836 2193 tablet_replica.cc:331] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc: stopping tablet replica
I20250115 02:54:09.187271 2193 raft_consensus.cc:2238] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:09.187646 2193 raft_consensus.cc:2267] T a8e4d4320eb14ab8a760b05d98fa71dd P b3c8272666bc45a29a8233aa9a9fe9fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:09.209681 2193 tablet_server.cc:195] TabletServer@127.2.36.67:0 shutdown complete.
I20250115 02:54:09.220256 2193 master.cc:537] Master@127.2.36.126:34885 shutting down...
I20250115 02:54:09.234691 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 1 LEADER]: Raft consensus shutting down.
I20250115 02:54:09.235603 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:09.235934 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P e3d0a417fcaa42fd8bf56e436ba3ec5c: stopping tablet replica
I20250115 02:54:09.254495 2193 master.cc:559] Master@127.2.36.126:34885 shutdown complete.
I20250115 02:54:09.266012 2193 master.cc:537] Master@127.2.36.125:43347 shutting down...
I20250115 02:54:09.283897 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:09.284366 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:09.284611 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 960aa779586e4ea4b7acbfcbaee1cff4: stopping tablet replica
I20250115 02:54:09.303385 2193 master.cc:559] Master@127.2.36.125:43347 shutdown complete.
I20250115 02:54:09.315629 2193 master.cc:537] Master@127.2.36.124:33697 shutting down...
I20250115 02:54:09.329283 2193 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 02:54:09.329753 2193 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 02:54:09.330001 2193 tablet_replica.cc:331] T 00000000000000000000000000000000 P 8ae7c243377948b4b16aca6bfeae4f8c: stopping tablet replica
I20250115 02:54:09.348934 2193 master.cc:559] Master@127.2.36.124:33697 shutdown complete.
[ SKIPPED ] RemoteKsckTest.RangeKeysInTabletSummaries (2538 ms)
[----------] 22 tests from RemoteKsckTest (123414 ms total)
[----------] Global test environment tear-down
[==========] 22 tests from 1 test suite ran. (123414 ms total)
[ PASSED ] 19 tests.
[ SKIPPED ] 2 tests, listed below:
[ SKIPPED ] RemoteKsckTest.TestChecksumSnapshotLastingLongerThanAHM
[ SKIPPED ] RemoteKsckTest.RangeKeysInTabletSummaries
[ FAILED ] 1 test, listed below:
[ FAILED ] RemoteKsckTest.TestChecksumSnapshotCurrentTimestamp
1 FAILED TEST
I20250115 02:54:09.384315 2193 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/ts_manager.cc:169: suppressed but not reported on 1 messages since previous log ~63 seconds ago
I20250115 02:54:09.384526 2193 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:4615: suppressed but not reported on 13 messages since previous log ~110 seconds ago
I20250115 02:54:09.384678 2193 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 945 messages since previous log ~110 seconds ago
I20250115 02:54:09.384994 2193 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2367: suppressed but not reported on 81 messages since previous log ~23 seconds ago
I20250115 02:54:09.385147 2193 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 14 messages since previous log ~5 seconds ago