[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20261116 08:57:50.756304 18840 thread.cc:642] rpc reactor (reactor) Time spent creating pthread: real 1.149s user 0.438s sys 0.710s
W20261116 08:57:50.756855 18840 thread.cc:609] rpc reactor (reactor) Time spent starting thread: real 1.150s user 0.438s sys 0.710s
I20261116 08:57:50.760985 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:36645
I20261116 08:57:50.762625 18840 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 08:57:50.763908 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:50.776328 18851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:50.777772 18852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:50.938938 18854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:50.939050 18840 server_base.cc:1061] running on GCE node
I20261116 08:57:50.940585 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:57:50.940785 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:50.940919 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819470940904 us; error 0 us; skew 500 ppm
I20261116 08:57:50.941571 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:50.945806 18840 webserver.cc:577] Webserver started at http://127.18.102.62:34139/ using document root <none> and password file <none>
I20261116 08:57:50.946408 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:50.946557 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:50.946884 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:50.953491 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "db132a6a80444829868c82a7105b27a8"
format_stamp: "Formatted at 2026-11-16 08:57:50 on dist-test-slave-qt0z"
I20261116 08:57:50.960704 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20261116 08:57:50.965029 18859 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:50.966012 18840 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20261116 08:57:50.966310 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "db132a6a80444829868c82a7105b27a8"
format_stamp: "Formatted at 2026-11-16 08:57:50 on dist-test-slave-qt0z"
I20261116 08:57:50.966578 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:50.987425 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:50.988713 18840 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 08:57:50.989099 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:51.055403 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:36645
I20261116 08:57:51.055475 18910 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:36645 every 8 connection(s)
I20261116 08:57:51.061277 18911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:51.081043 18911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8: Bootstrap starting.
I20261116 08:57:51.086697 18911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:51.088374 18911 log.cc:826] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8: Log is configured to *not* fsync() on all Append() calls
I20261116 08:57:51.092458 18911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8: No bootstrap required, opened a new log
I20261116 08:57:51.110925 18911 raft_consensus.cc:360] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db132a6a80444829868c82a7105b27a8" member_type: VOTER }
I20261116 08:57:51.111415 18911 raft_consensus.cc:386] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:57:51.111690 18911 raft_consensus.cc:748] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db132a6a80444829868c82a7105b27a8, State: Initialized, Role: FOLLOWER
I20261116 08:57:51.112469 18911 consensus_queue.cc:261] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db132a6a80444829868c82a7105b27a8" member_type: VOTER }
I20261116 08:57:51.112944 18911 raft_consensus.cc:400] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:57:51.113219 18911 raft_consensus.cc:499] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:57:51.113554 18911 raft_consensus.cc:3096] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:51.117246 18911 raft_consensus.cc:521] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db132a6a80444829868c82a7105b27a8" member_type: VOTER }
I20261116 08:57:51.117846 18911 leader_election.cc:304] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db132a6a80444829868c82a7105b27a8; no voters:
I20261116 08:57:51.119463 18911 leader_election.cc:290] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:57:51.119822 18914 raft_consensus.cc:2826] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:57:51.122355 18914 raft_consensus.cc:705] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [term 1 LEADER]: Becoming Leader. State: Replica: db132a6a80444829868c82a7105b27a8, State: Running, Role: LEADER
I20261116 08:57:51.123054 18914 consensus_queue.cc:238] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db132a6a80444829868c82a7105b27a8" member_type: VOTER }
I20261116 08:57:51.124085 18911 sys_catalog.cc:565] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:57:51.129529 18915 sys_catalog.cc:455] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db132a6a80444829868c82a7105b27a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db132a6a80444829868c82a7105b27a8" member_type: VOTER } }
I20261116 08:57:51.129879 18916 sys_catalog.cc:455] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db132a6a80444829868c82a7105b27a8. Latest consensus state: current_term: 1 leader_uuid: "db132a6a80444829868c82a7105b27a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db132a6a80444829868c82a7105b27a8" member_type: VOTER } }
I20261116 08:57:51.130108 18915 sys_catalog.cc:458] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [sys.catalog]: This master's current role is: LEADER
I20261116 08:57:51.130354 18916 sys_catalog.cc:458] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [sys.catalog]: This master's current role is: LEADER
I20261116 08:57:51.134095 18922 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:57:51.145910 18922 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:57:51.147694 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:57:51.160873 18922 catalog_manager.cc:1398] Generated new cluster ID: 0ba2aacb450648d39412d69fcfc9d4ea
I20261116 08:57:51.161113 18922 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:57:51.186272 18922 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:57:51.187650 18922 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:57:51.214047 18922 catalog_manager.cc:6221] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8: Generated new TSK 0
I20261116 08:57:51.214864 18922 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:57:51.278306 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:51.285933 18933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:51.287111 18934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:51.290532 18936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:51.290859 18840 server_base.cc:1061] running on GCE node
I20261116 08:57:51.291749 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:57:51.291985 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:51.292201 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819471292171 us; error 0 us; skew 500 ppm
I20261116 08:57:51.292816 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:51.296867 18840 webserver.cc:577] Webserver started at http://127.18.102.1:34177/ using document root <none> and password file <none>
I20261116 08:57:51.297380 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:51.297559 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:51.297847 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:51.298967 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "e2d3789911464b01847279f2a99b7802"
format_stamp: "Formatted at 2026-11-16 08:57:51 on dist-test-slave-qt0z"
I20261116 08:57:51.304340 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20261116 08:57:51.308571 18941 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:51.309366 18840 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20261116 08:57:51.309649 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "e2d3789911464b01847279f2a99b7802"
format_stamp: "Formatted at 2026-11-16 08:57:51 on dist-test-slave-qt0z"
I20261116 08:57:51.309914 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:51.335312 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:51.336581 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:51.339174 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:57:51.343422 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:57:51.343657 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:51.343946 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:57:51.344097 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:51.443959 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:40325
I20261116 08:57:51.444027 19003 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:40325 every 8 connection(s)
I20261116 08:57:51.450624 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:51.461200 19008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:51.475366 19004 heartbeater.cc:345] Connected to a master server at 127.18.102.62:36645
I20261116 08:57:51.475620 18840 server_base.cc:1061] running on GCE node
W20261116 08:57:51.475698 19011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:51.475893 19004 heartbeater.cc:462] Registering TS with master...
W20261116 08:57:51.464094 19009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:51.476963 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:57:51.477216 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:51.477231 19004 heartbeater.cc:508] Master 127.18.102.62:36645 requested a full tablet report, sending...
I20261116 08:57:51.477453 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819471477435 us; error 0 us; skew 500 ppm
I20261116 08:57:51.478063 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:51.480561 18876 ts_manager.cc:210] Registered new tserver with Master: e2d3789911464b01847279f2a99b7802 (127.18.102.1:40325)
I20261116 08:57:51.483417 18876 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48046
I20261116 08:57:51.483785 18840 webserver.cc:577] Webserver started at http://127.18.102.2:46837/ using document root <none> and password file <none>
I20261116 08:57:51.484294 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:51.484494 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:51.484797 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:51.486236 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/ts-1-root/instance:
uuid: "35235b3cf76247e5a9bc2c62c2790986"
format_stamp: "Formatted at 2026-11-16 08:57:51 on dist-test-slave-qt0z"
I20261116 08:57:51.491523 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20261116 08:57:51.495380 19016 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:51.496158 18840 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20261116 08:57:51.496435 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/ts-1-root
uuid: "35235b3cf76247e5a9bc2c62c2790986"
format_stamp: "Formatted at 2026-11-16 08:57:51 on dist-test-slave-qt0z"
I20261116 08:57:51.496737 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1794819469408847-18840-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:51.513527 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:51.514750 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:51.516912 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:57:51.519652 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:57:51.519870 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:51.520105 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:57:51.520270 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:51.586941 19004 heartbeater.cc:500] Master 127.18.102.62:36645 was elected leader, sending a full tablet report...
I20261116 08:57:51.613415 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.2:34705
I20261116 08:57:51.613525 19078 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.2:34705 every 8 connection(s)
I20261116 08:57:51.630595 19079 heartbeater.cc:345] Connected to a master server at 127.18.102.62:36645
I20261116 08:57:51.631016 19079 heartbeater.cc:462] Registering TS with master...
I20261116 08:57:51.631901 19079 heartbeater.cc:508] Master 127.18.102.62:36645 requested a full tablet report, sending...
I20261116 08:57:51.633814 18876 ts_manager.cc:210] Registered new tserver with Master: 35235b3cf76247e5a9bc2c62c2790986 (127.18.102.2:34705)
I20261116 08:57:51.634021 18840 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.015334296s
I20261116 08:57:51.635497 18876 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48052
I20261116 08:57:51.660211 18876 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
range_schema {
columns {
name: "key"
}
}
}
W20261116 08:57:51.677735 18876 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20261116 08:57:51.723534 18969 tablet_service.cc:1500] Processing CreateTablet for tablet 657e9ffe4d7a4788a0f3baf0f7b29bf7 (DEFAULT_TABLE table=test-table [id=19fc577ce2cc4104bbb87b98d178acd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261116 08:57:51.723634 18968 tablet_service.cc:1500] Processing CreateTablet for tablet 97ab6db2c8814384a8d5e1f04ea13bda (DEFAULT_TABLE table=test-table [id=19fc577ce2cc4104bbb87b98d178acd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261116 08:57:51.725037 18969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657e9ffe4d7a4788a0f3baf0f7b29bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:51.725656 18968 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97ab6db2c8814384a8d5e1f04ea13bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:51.729187 19044 tablet_service.cc:1500] Processing CreateTablet for tablet 657e9ffe4d7a4788a0f3baf0f7b29bf7 (DEFAULT_TABLE table=test-table [id=19fc577ce2cc4104bbb87b98d178acd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261116 08:57:51.729435 19043 tablet_service.cc:1500] Processing CreateTablet for tablet 97ab6db2c8814384a8d5e1f04ea13bda (DEFAULT_TABLE table=test-table [id=19fc577ce2cc4104bbb87b98d178acd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261116 08:57:51.730577 19044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657e9ffe4d7a4788a0f3baf0f7b29bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:51.732615 19043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97ab6db2c8814384a8d5e1f04ea13bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:51.738508 19079 heartbeater.cc:500] Master 127.18.102.62:36645 was elected leader, sending a full tablet report...
I20261116 08:57:51.751307 19093 tablet_bootstrap.cc:492] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986: Bootstrap starting.
I20261116 08:57:51.756740 19092 tablet_bootstrap.cc:492] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802: Bootstrap starting.
I20261116 08:57:51.762635 19093 tablet_bootstrap.cc:654] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:51.764236 19092 tablet_bootstrap.cc:654] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:51.770503 19093 tablet_bootstrap.cc:492] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986: No bootstrap required, opened a new log
I20261116 08:57:51.770888 19093 ts_tablet_manager.cc:1449] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.007s
I20261116 08:57:51.772943 19093 raft_consensus.cc:360] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:51.773432 19093 raft_consensus.cc:748] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35235b3cf76247e5a9bc2c62c2790986, State: Initialized, Role: FOLLOWER
I20261116 08:57:51.774099 19093 consensus_queue.cc:261] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:51.774713 19092 tablet_bootstrap.cc:492] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802: No bootstrap required, opened a new log
I20261116 08:57:51.775175 19092 ts_tablet_manager.cc:1449] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.005s
I20261116 08:57:51.777791 19093 ts_tablet_manager.cc:1480] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20261116 08:57:51.778723 19093 tablet_bootstrap.cc:492] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986: Bootstrap starting.
I20261116 08:57:51.778542 19092 raft_consensus.cc:360] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:51.779192 19092 raft_consensus.cc:748] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d3789911464b01847279f2a99b7802, State: Initialized, Role: FOLLOWER
I20261116 08:57:51.779760 19092 consensus_queue.cc:261] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:51.786278 19093 tablet_bootstrap.cc:654] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:51.791096 19092 ts_tablet_manager.cc:1480] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802: Time spent starting tablet: real 0.016s user 0.011s sys 0.000s
I20261116 08:57:51.792001 19092 tablet_bootstrap.cc:492] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802: Bootstrap starting.
I20261116 08:57:51.798630 19092 tablet_bootstrap.cc:654] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:51.804766 19093 tablet_bootstrap.cc:492] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986: No bootstrap required, opened a new log
I20261116 08:57:51.805331 19093 ts_tablet_manager.cc:1449] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.005s
I20261116 08:57:51.808140 19092 tablet_bootstrap.cc:492] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802: No bootstrap required, opened a new log
I20261116 08:57:51.808578 19092 ts_tablet_manager.cc:1449] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.007s
I20261116 08:57:51.808339 19093 raft_consensus.cc:360] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:51.808956 19093 raft_consensus.cc:748] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35235b3cf76247e5a9bc2c62c2790986, State: Initialized, Role: FOLLOWER
I20261116 08:57:51.809562 19093 consensus_queue.cc:261] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:51.811515 19093 ts_tablet_manager.cc:1480] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20261116 08:57:51.811237 19092 raft_consensus.cc:360] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:51.811959 19092 raft_consensus.cc:748] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d3789911464b01847279f2a99b7802, State: Initialized, Role: FOLLOWER
I20261116 08:57:51.812516 19092 consensus_queue.cc:261] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:51.814420 19092 ts_tablet_manager.cc:1480] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20261116 08:57:51.859449 18840 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20261116 08:57:51.859817 18840 raft_consensus.cc:425] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [term 0 FOLLOWER]: Emulating election...
I20261116 08:57:51.860062 18840 raft_consensus.cc:3096] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:51.863718 18840 raft_consensus.cc:705] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [term 1 LEADER]: Becoming Leader. State: Replica: 35235b3cf76247e5a9bc2c62c2790986, State: Running, Role: LEADER
I20261116 08:57:51.864346 18840 consensus_queue.cc:238] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:51.868080 18840 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
W20261116 08:57:51.869519 19080 tablet.cc:2441] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261116 08:57:51.878868 18874 catalog_manager.cc:5840] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35235b3cf76247e5a9bc2c62c2790986 (127.18.102.2). New cstate: current_term: 1 leader_uuid: "35235b3cf76247e5a9bc2c62c2790986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } health_report { overall_health: HEALTHY } } }
I20261116 08:57:51.891682 18979 raft_consensus.cc:3096] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:51.901175 18840 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20261116 08:57:51.901423 19096 consensus_queue.cc:1049] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261116 08:57:52.005853 18840 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20261116 08:57:52.006178 18840 raft_consensus.cc:425] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 0 FOLLOWER]: Emulating election...
I20261116 08:57:52.006458 18840 raft_consensus.cc:3096] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:52.012313 18840 raft_consensus.cc:705] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 1 LEADER]: Becoming Leader. State: Replica: 35235b3cf76247e5a9bc2c62c2790986, State: Running, Role: LEADER
I20261116 08:57:52.012985 18840 consensus_queue.cc:238] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:52.015343 18840 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261116 08:57:52.019656 18979 raft_consensus.cc:3096] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:52.019447 18874 catalog_manager.cc:5840] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35235b3cf76247e5a9bc2c62c2790986 (127.18.102.2). New cstate: current_term: 1 leader_uuid: "35235b3cf76247e5a9bc2c62c2790986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } health_report { overall_health: HEALTHY } } }
I20261116 08:57:52.020202 18840 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20261116 08:57:52.027557 19105 consensus_queue.cc:1049] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261116 08:57:52.324416 18840 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20261116 08:57:52.324803 18840 raft_consensus.cc:425] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 1 LEADER]: Emulating election...
I20261116 08:57:52.325124 18840 raft_consensus.cc:3091] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 1 LEADER]: Stepping down as leader of term 1
I20261116 08:57:52.325366 18840 raft_consensus.cc:748] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35235b3cf76247e5a9bc2c62c2790986, State: Running, Role: LEADER
I20261116 08:57:52.325928 18840 consensus_queue.cc:261] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:52.326916 18840 raft_consensus.cc:3096] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:57:52.333037 18840 raft_consensus.cc:705] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 2 LEADER]: Becoming Leader. State: Replica: 35235b3cf76247e5a9bc2c62c2790986, State: Running, Role: LEADER
I20261116 08:57:52.333789 18840 consensus_queue.cc:238] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:52.338421 18979 raft_consensus.cc:3096] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:57:52.339114 18840 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261116 08:57:52.342453 18875 catalog_manager.cc:5840] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "35235b3cf76247e5a9bc2c62c2790986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } health_report { overall_health: HEALTHY } } }
I20261116 08:57:52.345407 18979 raft_consensus.cc:1285] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 2 FOLLOWER]: Refusing update from remote peer 35235b3cf76247e5a9bc2c62c2790986: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261116 08:57:52.348033 19105 consensus_queue.cc:1049] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261116 08:57:52.351125 18840 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20261116 08:57:52.455439 18840 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20261116 08:57:52.455667 18840 raft_consensus.cc:425] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802 [term 1 FOLLOWER]: Emulating election...
I20261116 08:57:52.455924 18840 raft_consensus.cc:3096] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:57:52.459673 18840 raft_consensus.cc:705] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802 [term 2 LEADER]: Becoming Leader. State: Replica: e2d3789911464b01847279f2a99b7802, State: Running, Role: LEADER
I20261116 08:57:52.460325 18840 consensus_queue.cc:238] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:52.466740 18874 catalog_manager.cc:5840] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802 reported cstate change: term changed from 1 to 2, leader changed from 35235b3cf76247e5a9bc2c62c2790986 (127.18.102.2) to e2d3789911464b01847279f2a99b7802 (127.18.102.1). New cstate: current_term: 2 leader_uuid: "e2d3789911464b01847279f2a99b7802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } health_report { overall_health: UNKNOWN } } }
I20261116 08:57:52.468971 18840 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261116 08:57:52.475404 18840 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20261116 08:57:52.481236 19054 raft_consensus.cc:3091] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [term 1 LEADER]: Stepping down as leader of term 1
I20261116 08:57:52.481508 19054 raft_consensus.cc:748] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35235b3cf76247e5a9bc2c62c2790986, State: Running, Role: LEADER
I20261116 08:57:52.482088 19054 consensus_queue.cc:261] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:52.482820 19054 raft_consensus.cc:3096] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:57:52.487483 19054 raft_consensus.cc:1285] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [term 2 FOLLOWER]: Refusing update from remote peer e2d3789911464b01847279f2a99b7802: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261116 08:57:52.488878 19097 consensus_queue.cc:1049] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261116 08:57:52.779484 18840 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20261116 08:57:52.779721 18840 raft_consensus.cc:425] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 2 FOLLOWER]: Emulating election...
I20261116 08:57:52.779999 18840 raft_consensus.cc:3096] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:57:52.784915 18840 raft_consensus.cc:705] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 3 LEADER]: Becoming Leader. State: Replica: e2d3789911464b01847279f2a99b7802, State: Running, Role: LEADER
I20261116 08:57:52.785566 18840 consensus_queue.cc:238] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:52.787652 18840 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261116 08:57:52.788717 19054 raft_consensus.cc:3091] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 2 LEADER]: Stepping down as leader of term 2
I20261116 08:57:52.789047 19054 raft_consensus.cc:748] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 35235b3cf76247e5a9bc2c62c2790986, State: Running, Role: LEADER
I20261116 08:57:52.789614 19054 consensus_queue.cc:261] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:52.790473 19054 raft_consensus.cc:3096] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:57:52.790546 18874 catalog_manager.cc:5840] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 reported cstate change: term changed from 2 to 3, leader changed from 35235b3cf76247e5a9bc2c62c2790986 (127.18.102.2) to e2d3789911464b01847279f2a99b7802 (127.18.102.1). New cstate: current_term: 3 leader_uuid: "e2d3789911464b01847279f2a99b7802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } health_report { overall_health: UNKNOWN } } }
I20261116 08:57:52.797654 19054 raft_consensus.cc:1285] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 3 FOLLOWER]: Refusing update from remote peer e2d3789911464b01847279f2a99b7802: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20261116 08:57:52.799458 18840 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20261116 08:57:52.800716 19113 consensus_queue.cc:1049] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261116 08:57:52.903846 18840 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20261116 08:57:52.904043 18840 raft_consensus.cc:425] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 3 FOLLOWER]: Emulating election...
I20261116 08:57:52.904268 18840 raft_consensus.cc:3096] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 3 FOLLOWER]: Advancing to term 4
I20261116 08:57:52.907639 18840 raft_consensus.cc:705] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 4 LEADER]: Becoming Leader. State: Replica: 35235b3cf76247e5a9bc2c62c2790986, State: Running, Role: LEADER
I20261116 08:57:52.908216 18840 consensus_queue.cc:238] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:52.909454 18840 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261116 08:57:52.912722 18979 raft_consensus.cc:3091] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 3 LEADER]: Stepping down as leader of term 3
I20261116 08:57:52.913033 18979 raft_consensus.cc:748] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2d3789911464b01847279f2a99b7802, State: Running, Role: LEADER
I20261116 08:57:52.913183 18874 catalog_manager.cc:5840] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 reported cstate change: term changed from 3 to 4, leader changed from e2d3789911464b01847279f2a99b7802 (127.18.102.1) to 35235b3cf76247e5a9bc2c62c2790986 (127.18.102.2). New cstate: current_term: 4 leader_uuid: "35235b3cf76247e5a9bc2c62c2790986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } health_report { overall_health: HEALTHY } } }
I20261116 08:57:52.913552 18979 consensus_queue.cc:261] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 } } peers { permanent_uuid: "35235b3cf76247e5a9bc2c62c2790986" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 34705 } }
I20261116 08:57:52.914389 18979 raft_consensus.cc:3096] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 3 FOLLOWER]: Advancing to term 4
I20261116 08:57:52.916277 18840 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20261116 08:57:52.921446 18979 raft_consensus.cc:1285] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 4 FOLLOWER]: Refusing update from remote peer 35235b3cf76247e5a9bc2c62c2790986: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20261116 08:57:52.922827 19096 consensus_queue.cc:1049] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2d3789911464b01847279f2a99b7802" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261116 08:57:53.222114 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
I20261116 08:57:53.239588 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:57:53.240226 18840 tablet_replica.cc:333] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802: stopping tablet replica
I20261116 08:57:53.240813 18840 raft_consensus.cc:2261] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802 [term 2 LEADER]: Raft consensus shutting down.
I20261116 08:57:53.241400 18840 raft_consensus.cc:2290] T 97ab6db2c8814384a8d5e1f04ea13bda P e2d3789911464b01847279f2a99b7802 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261116 08:57:53.243818 18840 tablet_replica.cc:333] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802: stopping tablet replica
I20261116 08:57:53.244268 18840 raft_consensus.cc:2261] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 4 FOLLOWER]: Raft consensus shutting down.
I20261116 08:57:53.244652 18840 raft_consensus.cc:2290] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P e2d3789911464b01847279f2a99b7802 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261116 08:57:53.326743 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
I20261116 08:57:53.336175 18840 tablet_server.cc:179] TabletServer@127.18.102.2:0 shutting down...
I20261116 08:57:53.352543 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:57:53.353094 18840 tablet_replica.cc:333] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986: stopping tablet replica
I20261116 08:57:53.353598 18840 raft_consensus.cc:2261] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20261116 08:57:53.353935 18840 raft_consensus.cc:2290] T 97ab6db2c8814384a8d5e1f04ea13bda P 35235b3cf76247e5a9bc2c62c2790986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261116 08:57:53.355610 18840 tablet_replica.cc:333] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986: stopping tablet replica
I20261116 08:57:53.355973 18840 raft_consensus.cc:2261] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 4 LEADER]: Raft consensus shutting down.
I20261116 08:57:53.356410 18840 raft_consensus.cc:2290] T 657e9ffe4d7a4788a0f3baf0f7b29bf7 P 35235b3cf76247e5a9bc2c62c2790986 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261116 08:57:53.363981 18840 tablet_server.cc:196] TabletServer@127.18.102.2:0 shutdown complete.
I20261116 08:57:53.371510 18840 master.cc:585] Master@127.18.102.62:36645 shutting down...
I20261116 08:57:53.445178 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:57:53.445690 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:57:53.446002 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P db132a6a80444829868c82a7105b27a8: stopping tablet replica
I20261116 08:57:53.464373 18840 master.cc:607] Master@127.18.102.62:36645 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (3925 ms)
[ RUN ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20261116 08:57:53.488446 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:46133
I20261116 08:57:53.489398 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:53.493897 19120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:53.494843 19121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:53.496681 18840 server_base.cc:1061] running on GCE node
W20261116 08:57:53.497480 19123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:53.498221 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:57:53.498389 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:53.498497 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819473498487 us; error 0 us; skew 500 ppm
I20261116 08:57:53.498893 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:53.500885 18840 webserver.cc:577] Webserver started at http://127.18.102.62:45821/ using document root <none> and password file <none>
I20261116 08:57:53.501256 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:53.501392 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:53.501592 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:53.502475 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "68cb83830d804bf5966be1c6d8818e5f"
format_stamp: "Formatted at 2026-11-16 08:57:53 on dist-test-slave-qt0z"
I20261116 08:57:53.506541 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20261116 08:57:53.509426 19128 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:53.510118 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261116 08:57:53.510380 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "68cb83830d804bf5966be1c6d8818e5f"
format_stamp: "Formatted at 2026-11-16 08:57:53 on dist-test-slave-qt0z"
I20261116 08:57:53.510697 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:53.533382 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:53.534293 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:53.565613 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:46133
I20261116 08:57:53.565699 19179 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:46133 every 8 connection(s)
I20261116 08:57:53.569172 19180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:53.577677 19180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f: Bootstrap starting.
I20261116 08:57:53.581408 19180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:53.584698 19180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f: No bootstrap required, opened a new log
I20261116 08:57:53.586522 19180 raft_consensus.cc:360] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cb83830d804bf5966be1c6d8818e5f" member_type: VOTER }
I20261116 08:57:53.586885 19180 raft_consensus.cc:386] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:57:53.587128 19180 raft_consensus.cc:748] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68cb83830d804bf5966be1c6d8818e5f, State: Initialized, Role: FOLLOWER
I20261116 08:57:53.587654 19180 consensus_queue.cc:261] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cb83830d804bf5966be1c6d8818e5f" member_type: VOTER }
I20261116 08:57:53.587998 19180 raft_consensus.cc:400] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:57:53.588227 19180 raft_consensus.cc:499] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:57:53.588491 19180 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:53.591809 19180 raft_consensus.cc:521] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cb83830d804bf5966be1c6d8818e5f" member_type: VOTER }
I20261116 08:57:53.592288 19180 leader_election.cc:304] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68cb83830d804bf5966be1c6d8818e5f; no voters:
I20261116 08:57:53.593214 19180 leader_election.cc:290] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:57:53.593575 19183 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:57:53.594882 19183 raft_consensus.cc:705] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [term 1 LEADER]: Becoming Leader. State: Replica: 68cb83830d804bf5966be1c6d8818e5f, State: Running, Role: LEADER
I20261116 08:57:53.595525 19183 consensus_queue.cc:238] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cb83830d804bf5966be1c6d8818e5f" member_type: VOTER }
I20261116 08:57:53.595916 19180 sys_catalog.cc:565] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:57:53.598037 19184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68cb83830d804bf5966be1c6d8818e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cb83830d804bf5966be1c6d8818e5f" member_type: VOTER } }
I20261116 08:57:53.598052 19185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68cb83830d804bf5966be1c6d8818e5f. Latest consensus state: current_term: 1 leader_uuid: "68cb83830d804bf5966be1c6d8818e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cb83830d804bf5966be1c6d8818e5f" member_type: VOTER } }
I20261116 08:57:53.598817 19184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [sys.catalog]: This master's current role is: LEADER
I20261116 08:57:53.598839 19185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [sys.catalog]: This master's current role is: LEADER
I20261116 08:57:53.601234 19189 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:57:53.608723 19189 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:57:53.609525 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:57:53.616365 19189 catalog_manager.cc:1398] Generated new cluster ID: 6e453bc7ef944d2f9acb80bdcf764c6e
I20261116 08:57:53.616623 19189 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:57:53.628342 19189 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:57:53.629355 19189 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:57:53.642834 19189 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f: Generated new TSK 0
I20261116 08:57:53.643376 19189 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:57:53.675493 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:53.680652 19201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:53.681803 19202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:53.683051 19204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:53.683185 18840 server_base.cc:1061] running on GCE node
I20261116 08:57:53.683967 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:57:53.684122 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:53.684228 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819473684218 us; error 0 us; skew 500 ppm
I20261116 08:57:53.684607 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:53.686640 18840 webserver.cc:577] Webserver started at http://127.18.102.1:43173/ using document root <none> and password file <none>
I20261116 08:57:53.687043 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:53.687233 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:53.687424 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:53.688298 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "c7d3e40244d04e038d81dd94297d805e"
format_stamp: "Formatted at 2026-11-16 08:57:53 on dist-test-slave-qt0z"
I20261116 08:57:53.691977 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20261116 08:57:53.694746 19209 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:53.695472 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20261116 08:57:53.695703 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "c7d3e40244d04e038d81dd94297d805e"
format_stamp: "Formatted at 2026-11-16 08:57:53 on dist-test-slave-qt0z"
I20261116 08:57:53.695912 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:53.703243 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:53.703974 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:53.705258 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:57:53.707074 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:57:53.707284 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:53.707441 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:57:53.707551 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:53.739668 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:36615
I20261116 08:57:53.739753 19271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:36615 every 8 connection(s)
I20261116 08:57:53.743958 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:53.749419 19276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:53.751541 19277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:53.752627 18840 server_base.cc:1061] running on GCE node
W20261116 08:57:53.754158 19279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:53.754698 19272 heartbeater.cc:345] Connected to a master server at 127.18.102.62:46133
I20261116 08:57:53.755028 19272 heartbeater.cc:462] Registering TS with master...
I20261116 08:57:53.755041 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:57:53.755427 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:53.755601 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819473755572 us; error 0 us; skew 500 ppm
I20261116 08:57:53.755906 19272 heartbeater.cc:508] Master 127.18.102.62:46133 requested a full tablet report, sending...
I20261116 08:57:53.756232 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:53.757934 19145 ts_manager.cc:210] Registered new tserver with Master: c7d3e40244d04e038d81dd94297d805e (127.18.102.1:36615)
I20261116 08:57:53.759523 19145 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35730
I20261116 08:57:53.760541 18840 webserver.cc:577] Webserver started at http://127.18.102.2:42029/ using document root <none> and password file <none>
I20261116 08:57:53.760972 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:53.761158 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:53.761406 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:53.762400 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/ts-1-root/instance:
uuid: "11092e2204a24964a5117abe057f691b"
format_stamp: "Formatted at 2026-11-16 08:57:53 on dist-test-slave-qt0z"
I20261116 08:57:53.766386 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20261116 08:57:53.769253 19284 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:53.770020 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261116 08:57:53.770231 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/ts-1-root
uuid: "11092e2204a24964a5117abe057f691b"
format_stamp: "Formatted at 2026-11-16 08:57:53 on dist-test-slave-qt0z"
I20261116 08:57:53.770430 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1794819469408847-18840-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:53.811677 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:53.812775 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:53.814054 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:57:53.816157 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:57:53.816331 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:53.816630 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:57:53.816788 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:53.852046 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.2:33491
I20261116 08:57:53.852138 19346 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.2:33491 every 8 connection(s)
I20261116 08:57:53.862027 19272 heartbeater.cc:500] Master 127.18.102.62:46133 was elected leader, sending a full tablet report...
I20261116 08:57:53.864722 19347 heartbeater.cc:345] Connected to a master server at 127.18.102.62:46133
I20261116 08:57:53.865080 19347 heartbeater.cc:462] Registering TS with master...
I20261116 08:57:53.865847 19347 heartbeater.cc:508] Master 127.18.102.62:46133 requested a full tablet report, sending...
I20261116 08:57:53.867532 19145 ts_manager.cc:210] Registered new tserver with Master: 11092e2204a24964a5117abe057f691b (127.18.102.2:33491)
I20261116 08:57:53.868388 18840 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.013104966s
I20261116 08:57:53.868721 19145 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35742
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:422: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20261116 08:57:53.886873 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
I20261116 08:57:53.900810 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:57:53.915692 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
I20261116 08:57:53.921663 18840 tablet_server.cc:179] TabletServer@127.18.102.2:0 shutting down...
I20261116 08:57:53.935863 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:57:53.950547 18840 tablet_server.cc:196] TabletServer@127.18.102.2:0 shutdown complete.
I20261116 08:57:53.956746 18840 master.cc:585] Master@127.18.102.62:46133 shutting down...
I20261116 08:57:53.972513 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:57:53.972926 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:57:53.973238 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P 68cb83830d804bf5966be1c6d8818e5f: stopping tablet replica
I20261116 08:57:53.990363 18840 master.cc:607] Master@127.18.102.62:46133 shutdown complete.
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (522 ms)
[ RUN ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20261116 08:57:54.010140 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:41175
I20261116 08:57:54.011066 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:54.015530 19360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:54.016732 19361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:54.018106 19363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:54.018782 18840 server_base.cc:1061] running on GCE node
I20261116 08:57:54.019524 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:57:54.019694 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:54.019881 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819474019850 us; error 0 us; skew 500 ppm
I20261116 08:57:54.020381 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:54.022377 18840 webserver.cc:577] Webserver started at http://127.18.102.62:35553/ using document root <none> and password file <none>
I20261116 08:57:54.022830 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:54.023005 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:54.023300 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:54.024281 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "5eca0047534b4379890bc8a15813c1d2"
format_stamp: "Formatted at 2026-11-16 08:57:54 on dist-test-slave-qt0z"
I20261116 08:57:54.028187 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261116 08:57:54.031104 19368 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:54.031817 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261116 08:57:54.032084 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "5eca0047534b4379890bc8a15813c1d2"
format_stamp: "Formatted at 2026-11-16 08:57:54 on dist-test-slave-qt0z"
I20261116 08:57:54.032353 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:54.039687 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:54.040544 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:54.072171 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:41175
I20261116 08:57:54.072257 19419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:41175 every 8 connection(s)
I20261116 08:57:54.075515 19420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:54.083738 19420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2: Bootstrap starting.
I20261116 08:57:54.087486 19420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:54.090605 19420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2: No bootstrap required, opened a new log
I20261116 08:57:54.092315 19420 raft_consensus.cc:360] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eca0047534b4379890bc8a15813c1d2" member_type: VOTER }
I20261116 08:57:54.092644 19420 raft_consensus.cc:386] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:57:54.092835 19420 raft_consensus.cc:748] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eca0047534b4379890bc8a15813c1d2, State: Initialized, Role: FOLLOWER
I20261116 08:57:54.093348 19420 consensus_queue.cc:261] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eca0047534b4379890bc8a15813c1d2" member_type: VOTER }
I20261116 08:57:54.093722 19420 raft_consensus.cc:400] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:57:54.093891 19420 raft_consensus.cc:499] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:57:54.094097 19420 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:54.097447 19420 raft_consensus.cc:521] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eca0047534b4379890bc8a15813c1d2" member_type: VOTER }
I20261116 08:57:54.097836 19420 leader_election.cc:304] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eca0047534b4379890bc8a15813c1d2; no voters:
I20261116 08:57:54.098722 19420 leader_election.cc:290] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:57:54.098999 19423 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:57:54.100157 19423 raft_consensus.cc:705] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [term 1 LEADER]: Becoming Leader. State: Replica: 5eca0047534b4379890bc8a15813c1d2, State: Running, Role: LEADER
I20261116 08:57:54.100721 19423 consensus_queue.cc:238] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eca0047534b4379890bc8a15813c1d2" member_type: VOTER }
I20261116 08:57:54.101339 19420 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:57:54.103323 19425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eca0047534b4379890bc8a15813c1d2. Latest consensus state: current_term: 1 leader_uuid: "5eca0047534b4379890bc8a15813c1d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eca0047534b4379890bc8a15813c1d2" member_type: VOTER } }
I20261116 08:57:54.103370 19424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eca0047534b4379890bc8a15813c1d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eca0047534b4379890bc8a15813c1d2" member_type: VOTER } }
I20261116 08:57:54.103945 19425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [sys.catalog]: This master's current role is: LEADER
I20261116 08:57:54.104046 19424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [sys.catalog]: This master's current role is: LEADER
I20261116 08:57:54.106182 19429 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:57:54.110481 19429 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:57:54.116014 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:57:54.117781 19429 catalog_manager.cc:1398] Generated new cluster ID: c0ba89ef44a844148270b18596e0b2df
I20261116 08:57:54.118012 19429 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:57:54.132483 19429 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:57:54.133623 19429 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:57:54.144706 19429 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2: Generated new TSK 0
I20261116 08:57:54.145185 19429 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:57:54.149781 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:54.154891 19441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:54.155802 19442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:54.157065 19444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:54.157378 18840 server_base.cc:1061] running on GCE node
I20261116 08:57:54.158070 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:57:54.158237 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:54.158339 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819474158330 us; error 0 us; skew 500 ppm
I20261116 08:57:54.158742 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:54.160687 18840 webserver.cc:577] Webserver started at http://127.18.102.1:36691/ using document root <none> and password file <none>
I20261116 08:57:54.161072 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:54.161244 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:54.161432 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:54.162324 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "6351a376639740f58a14dbc61eb5fed4"
format_stamp: "Formatted at 2026-11-16 08:57:54 on dist-test-slave-qt0z"
I20261116 08:57:54.166126 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261116 08:57:54.168815 19449 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:54.169423 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261116 08:57:54.169696 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "6351a376639740f58a14dbc61eb5fed4"
format_stamp: "Formatted at 2026-11-16 08:57:54 on dist-test-slave-qt0z"
I20261116 08:57:54.169957 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:54.190539 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:54.191517 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:54.192790 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:57:54.194705 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:57:54.194909 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:54.195132 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:57:54.195379 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20261116 08:57:54.231248 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:37013
I20261116 08:57:54.231338 19511 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:37013 every 8 connection(s)
I20261116 08:57:54.236071 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:54.242727 19516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:54.243484 19517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:54.245620 19519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:54.246697 18840 server_base.cc:1061] running on GCE node
I20261116 08:57:54.247602 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20261116 08:57:54.247664 19512 heartbeater.cc:345] Connected to a master server at 127.18.102.62:41175
W20261116 08:57:54.247910 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:54.248116 19512 heartbeater.cc:462] Registering TS with master...
I20261116 08:57:54.248174 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819474248155 us; error 0 us; skew 500 ppm
I20261116 08:57:54.248826 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:54.248919 19512 heartbeater.cc:508] Master 127.18.102.62:41175 requested a full tablet report, sending...
I20261116 08:57:54.250998 19385 ts_manager.cc:210] Registered new tserver with Master: 6351a376639740f58a14dbc61eb5fed4 (127.18.102.1:37013)
I20261116 08:57:54.251426 18840 webserver.cc:577] Webserver started at http://127.18.102.2:36537/ using document root <none> and password file <none>
I20261116 08:57:54.251914 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:54.252125 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:54.252388 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:54.253180 19385 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48166
I20261116 08:57:54.253435 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/ts-1-root/instance:
uuid: "81b88e867162449ba02f6575a1ebfdd0"
format_stamp: "Formatted at 2026-11-16 08:57:54 on dist-test-slave-qt0z"
I20261116 08:57:54.258147 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20261116 08:57:54.261179 19524 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:54.261881 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261116 08:57:54.262162 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/ts-1-root
uuid: "81b88e867162449ba02f6575a1ebfdd0"
format_stamp: "Formatted at 2026-11-16 08:57:54 on dist-test-slave-qt0z"
I20261116 08:57:54.262444 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1794819469408847-18840-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:54.294034 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:54.295032 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:54.296399 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:57:54.298460 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:57:54.298687 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:54.298940 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:57:54.299117 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:54.333683 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.2:33577
I20261116 08:57:54.333784 19586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.2:33577 every 8 connection(s)
I20261116 08:57:54.345522 19587 heartbeater.cc:345] Connected to a master server at 127.18.102.62:41175
I20261116 08:57:54.345875 19587 heartbeater.cc:462] Registering TS with master...
I20261116 08:57:54.346539 19587 heartbeater.cc:508] Master 127.18.102.62:41175 requested a full tablet report, sending...
I20261116 08:57:54.348201 19385 ts_manager.cc:210] Registered new tserver with Master: 81b88e867162449ba02f6575a1ebfdd0 (127.18.102.2:33577)
I20261116 08:57:54.348598 18840 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.011913425s
I20261116 08:57:54.349682 19385 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48176
I20261116 08:57:54.355849 19512 heartbeater.cc:500] Master 127.18.102.62:41175 was elected leader, sending a full tablet report...
I20261116 08:57:54.369287 19385 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20261116 08:57:54.370529 19385 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20261116 08:57:54.411980 19477 tablet_service.cc:1500] Processing CreateTablet for tablet ee22685f630d421dae897c7ff0144adc (DEFAULT_TABLE table=test-table [id=bd7f7f1094f34670a7d4c038a1c5ed28]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 08:57:54.412714 19552 tablet_service.cc:1500] Processing CreateTablet for tablet ee22685f630d421dae897c7ff0144adc (DEFAULT_TABLE table=test-table [id=bd7f7f1094f34670a7d4c038a1c5ed28]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 08:57:54.413259 19477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee22685f630d421dae897c7ff0144adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:54.413875 19552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee22685f630d421dae897c7ff0144adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:54.424018 19600 tablet_bootstrap.cc:492] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4: Bootstrap starting.
I20261116 08:57:54.424507 19601 tablet_bootstrap.cc:492] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0: Bootstrap starting.
I20261116 08:57:54.429118 19600 tablet_bootstrap.cc:654] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:54.429494 19601 tablet_bootstrap.cc:654] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:54.432826 19600 tablet_bootstrap.cc:492] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4: No bootstrap required, opened a new log
I20261116 08:57:54.433168 19600 ts_tablet_manager.cc:1449] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20261116 08:57:54.433241 19601 tablet_bootstrap.cc:492] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0: No bootstrap required, opened a new log
I20261116 08:57:54.433670 19601 ts_tablet_manager.cc:1449] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.009s
I20261116 08:57:54.435536 19600 raft_consensus.cc:360] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6351a376639740f58a14dbc61eb5fed4" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37013 } } peers { permanent_uuid: "81b88e867162449ba02f6575a1ebfdd0" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 33577 } }
I20261116 08:57:54.435984 19601 raft_consensus.cc:360] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6351a376639740f58a14dbc61eb5fed4" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37013 } } peers { permanent_uuid: "81b88e867162449ba02f6575a1ebfdd0" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 33577 } }
I20261116 08:57:54.436175 19600 raft_consensus.cc:748] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6351a376639740f58a14dbc61eb5fed4, State: Initialized, Role: FOLLOWER
I20261116 08:57:54.436568 19601 raft_consensus.cc:748] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b88e867162449ba02f6575a1ebfdd0, State: Initialized, Role: FOLLOWER
I20261116 08:57:54.436758 19600 consensus_queue.cc:261] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6351a376639740f58a14dbc61eb5fed4" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37013 } } peers { permanent_uuid: "81b88e867162449ba02f6575a1ebfdd0" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 33577 } }
I20261116 08:57:54.437029 19601 consensus_queue.cc:261] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6351a376639740f58a14dbc61eb5fed4" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37013 } } peers { permanent_uuid: "81b88e867162449ba02f6575a1ebfdd0" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 33577 } }
I20261116 08:57:54.439172 19587 heartbeater.cc:500] Master 127.18.102.62:41175 was elected leader, sending a full tablet report...
I20261116 08:57:54.439726 19601 ts_tablet_manager.cc:1480] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20261116 08:57:54.439980 19600 ts_tablet_manager.cc:1480] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20261116 08:57:54.472398 18840 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20261116 08:57:54.472621 18840 raft_consensus.cc:425] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 0 FOLLOWER]: Emulating election...
I20261116 08:57:54.472872 18840 raft_consensus.cc:3096] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:54.476558 18840 raft_consensus.cc:705] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 1 LEADER]: Becoming Leader. State: Replica: 6351a376639740f58a14dbc61eb5fed4, State: Running, Role: LEADER
W20261116 08:57:54.476881 19512 heartbeater.cc:648] Failed to heartbeat to 127.18.102.62:41175 (0 consecutive failures): IO error: failing all heartbeats for tests
I20261116 08:57:54.477252 18840 consensus_queue.cc:238] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6351a376639740f58a14dbc61eb5fed4" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37013 } } peers { permanent_uuid: "81b88e867162449ba02f6575a1ebfdd0" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 33577 } }
I20261116 08:57:54.479348 18840 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261116 08:57:54.488451 19562 raft_consensus.cc:3096] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:54.495857 19604 consensus_queue.cc:1049] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b88e867162449ba02f6575a1ebfdd0" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 08:57:54.501909 18840 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20261116 08:57:54.502140 18840 raft_consensus.cc:425] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 1 LEADER]: Emulating election...
I20261116 08:57:54.502362 18840 raft_consensus.cc:3091] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 1 LEADER]: Stepping down as leader of term 1
I20261116 08:57:54.502636 18840 raft_consensus.cc:748] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6351a376639740f58a14dbc61eb5fed4, State: Running, Role: LEADER
I20261116 08:57:54.503206 18840 consensus_queue.cc:261] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6351a376639740f58a14dbc61eb5fed4" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37013 } } peers { permanent_uuid: "81b88e867162449ba02f6575a1ebfdd0" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 33577 } }
I20261116 08:57:54.504174 18840 raft_consensus.cc:3096] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:57:54.507612 18840 raft_consensus.cc:705] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 2 LEADER]: Becoming Leader. State: Replica: 6351a376639740f58a14dbc61eb5fed4, State: Running, Role: LEADER
I20261116 08:57:54.508129 18840 consensus_queue.cc:238] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6351a376639740f58a14dbc61eb5fed4" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37013 } } peers { permanent_uuid: "81b88e867162449ba02f6575a1ebfdd0" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 33577 } }
I20261116 08:57:54.509297 18840 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261116 08:57:54.510856 19562 raft_consensus.cc:3096] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:57:54.514770 19562 raft_consensus.cc:1285] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0 [term 2 FOLLOWER]: Refusing update from remote peer 6351a376639740f58a14dbc61eb5fed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261116 08:57:54.516124 19604 consensus_queue.cc:1049] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b88e867162449ba02f6575a1ebfdd0" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20261116 08:57:54.518483 18840 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20261116 08:57:54.623294 18840 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20261116 08:57:54.623548 18840 raft_consensus.cc:425] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 2 LEADER]: Emulating election...
I20261116 08:57:54.623762 18840 raft_consensus.cc:3091] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 2 LEADER]: Stepping down as leader of term 2
I20261116 08:57:54.623962 18840 raft_consensus.cc:748] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6351a376639740f58a14dbc61eb5fed4, State: Running, Role: LEADER
I20261116 08:57:54.624367 18840 consensus_queue.cc:261] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6351a376639740f58a14dbc61eb5fed4" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37013 } } peers { permanent_uuid: "81b88e867162449ba02f6575a1ebfdd0" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 33577 } }
I20261116 08:57:54.624966 18840 raft_consensus.cc:3096] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:57:54.628666 18840 raft_consensus.cc:705] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 3 LEADER]: Becoming Leader. State: Replica: 6351a376639740f58a14dbc61eb5fed4, State: Running, Role: LEADER
I20261116 08:57:54.629221 18840 consensus_queue.cc:238] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6351a376639740f58a14dbc61eb5fed4" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37013 } } peers { permanent_uuid: "81b88e867162449ba02f6575a1ebfdd0" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 33577 } }
I20261116 08:57:54.630550 18840 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261116 08:57:54.632140 19562 raft_consensus.cc:3096] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0 [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:57:54.636785 19562 raft_consensus.cc:1285] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0 [term 3 FOLLOWER]: Refusing update from remote peer 6351a376639740f58a14dbc61eb5fed4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20261116 08:57:54.638226 18840 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20261116 08:57:54.638031 19609 consensus_queue.cc:1049] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b88e867162449ba02f6575a1ebfdd0" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261116 08:57:54.745002 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
I20261116 08:57:54.761263 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:57:54.762058 18840 tablet_replica.cc:333] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4: stopping tablet replica
I20261116 08:57:54.762521 18840 raft_consensus.cc:2261] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 3 LEADER]: Raft consensus shutting down.
I20261116 08:57:54.763016 18840 raft_consensus.cc:2290] T ee22685f630d421dae897c7ff0144adc P 6351a376639740f58a14dbc61eb5fed4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20261116 08:57:54.781647 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
I20261116 08:57:54.789178 18840 tablet_server.cc:179] TabletServer@127.18.102.2:0 shutting down...
I20261116 08:57:54.804834 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:57:54.805588 18840 tablet_replica.cc:333] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0: stopping tablet replica
I20261116 08:57:54.806022 18840 raft_consensus.cc:2261] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20261116 08:57:54.806380 18840 raft_consensus.cc:2290] T ee22685f630d421dae897c7ff0144adc P 81b88e867162449ba02f6575a1ebfdd0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20261116 08:57:54.824620 18840 tablet_server.cc:196] TabletServer@127.18.102.2:0 shutdown complete.
I20261116 08:57:54.831743 18840 master.cc:585] Master@127.18.102.62:41175 shutting down...
I20261116 08:57:54.844426 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:57:54.844926 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:57:54.845216 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P 5eca0047534b4379890bc8a15813c1d2: stopping tablet replica
I20261116 08:57:54.862579 18840 master.cc:607] Master@127.18.102.62:41175 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (874 ms)
[----------] 3 tests from LeadershipChangeReportingTest (5322 ms total)
[----------] 4 tests from TsTabletManagerITest
[ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20261116 08:57:54.885917 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:42427
I20261116 08:57:54.886947 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:54.891368 19617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:54.892668 19618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:54.894066 19620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:54.894714 18840 server_base.cc:1061] running on GCE node
I20261116 08:57:54.895758 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:57:54.895982 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:54.896198 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819474896182 us; error 0 us; skew 500 ppm
I20261116 08:57:54.896720 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:54.898792 18840 webserver.cc:577] Webserver started at http://127.18.102.62:42449/ using document root <none> and password file <none>
I20261116 08:57:54.899366 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:54.899575 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:54.899850 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:54.900933 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "76b58a7bde4a4164a118eb3b9592b62a"
format_stamp: "Formatted at 2026-11-16 08:57:54 on dist-test-slave-qt0z"
I20261116 08:57:54.905490 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20261116 08:57:54.908739 19625 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:54.909629 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261116 08:57:54.909868 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "76b58a7bde4a4164a118eb3b9592b62a"
format_stamp: "Formatted at 2026-11-16 08:57:54 on dist-test-slave-qt0z"
I20261116 08:57:54.910110 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:54.943856 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:54.944820 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:54.979367 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:42427
I20261116 08:57:54.979449 19676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:42427 every 8 connection(s)
I20261116 08:57:54.982959 19677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:54.991891 19677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a: Bootstrap starting.
I20261116 08:57:54.995743 19677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:54.999292 19677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a: No bootstrap required, opened a new log
I20261116 08:57:55.001031 19677 raft_consensus.cc:360] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b58a7bde4a4164a118eb3b9592b62a" member_type: VOTER }
I20261116 08:57:55.001369 19677 raft_consensus.cc:386] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:57:55.001547 19677 raft_consensus.cc:748] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b58a7bde4a4164a118eb3b9592b62a, State: Initialized, Role: FOLLOWER
I20261116 08:57:55.002079 19677 consensus_queue.cc:261] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b58a7bde4a4164a118eb3b9592b62a" member_type: VOTER }
I20261116 08:57:55.002457 19677 raft_consensus.cc:400] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:57:55.002641 19677 raft_consensus.cc:499] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:57:55.002840 19677 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:55.006508 19677 raft_consensus.cc:521] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b58a7bde4a4164a118eb3b9592b62a" member_type: VOTER }
I20261116 08:57:55.006908 19677 leader_election.cc:304] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76b58a7bde4a4164a118eb3b9592b62a; no voters:
I20261116 08:57:55.007891 19677 leader_election.cc:290] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:57:55.008199 19680 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:57:55.009379 19680 raft_consensus.cc:705] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [term 1 LEADER]: Becoming Leader. State: Replica: 76b58a7bde4a4164a118eb3b9592b62a, State: Running, Role: LEADER
I20261116 08:57:55.009975 19680 consensus_queue.cc:238] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b58a7bde4a4164a118eb3b9592b62a" member_type: VOTER }
I20261116 08:57:55.010515 19677 sys_catalog.cc:565] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:57:55.012708 19681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76b58a7bde4a4164a118eb3b9592b62a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b58a7bde4a4164a118eb3b9592b62a" member_type: VOTER } }
I20261116 08:57:55.012764 19682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76b58a7bde4a4164a118eb3b9592b62a. Latest consensus state: current_term: 1 leader_uuid: "76b58a7bde4a4164a118eb3b9592b62a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b58a7bde4a4164a118eb3b9592b62a" member_type: VOTER } }
I20261116 08:57:55.013367 19681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [sys.catalog]: This master's current role is: LEADER
I20261116 08:57:55.013403 19682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [sys.catalog]: This master's current role is: LEADER
I20261116 08:57:55.016038 19686 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:57:55.020788 19686 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:57:55.024916 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:57:55.029003 19686 catalog_manager.cc:1398] Generated new cluster ID: 43863743846b42828d27be6fbc055005
I20261116 08:57:55.029210 19686 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:57:55.056104 19686 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:57:55.057430 19686 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:57:55.073256 19686 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a: Generated new TSK 0
I20261116 08:57:55.073725 19686 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:57:55.090696 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:55.095960 19698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:55.097095 19699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:55.098351 19701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:55.098737 18840 server_base.cc:1061] running on GCE node
I20261116 08:57:55.099507 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:57:55.099670 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:55.099782 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819475099771 us; error 0 us; skew 500 ppm
I20261116 08:57:55.100229 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:55.102272 18840 webserver.cc:577] Webserver started at http://127.18.102.1:41257/ using document root <none> and password file <none>
I20261116 08:57:55.102691 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:55.102852 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:55.103029 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:55.103986 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "71498a460e3d4967965f341259b194f7"
format_stamp: "Formatted at 2026-11-16 08:57:55 on dist-test-slave-qt0z"
I20261116 08:57:55.107939 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20261116 08:57:55.110860 19706 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:55.111716 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20261116 08:57:55.112022 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "71498a460e3d4967965f341259b194f7"
format_stamp: "Formatted at 2026-11-16 08:57:55 on dist-test-slave-qt0z"
I20261116 08:57:55.112293 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:55.128620 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:55.129572 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:55.130884 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:57:55.133029 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:57:55.133251 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:55.133486 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:57:55.133651 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:55.168671 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:45883
I20261116 08:57:55.168761 19768 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:45883 every 8 connection(s)
I20261116 08:57:55.173282 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:55.179610 19773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:55.181028 19774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:55.183740 18840 server_base.cc:1061] running on GCE node
W20261116 08:57:55.183702 19776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:55.184796 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:57:55.185055 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:55.185168 19769 heartbeater.cc:345] Connected to a master server at 127.18.102.62:42427
I20261116 08:57:55.185305 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819475185285 us; error 0 us; skew 500 ppm
I20261116 08:57:55.185520 19769 heartbeater.cc:462] Registering TS with master...
I20261116 08:57:55.185990 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:55.186311 19769 heartbeater.cc:508] Master 127.18.102.62:42427 requested a full tablet report, sending...
I20261116 08:57:55.188514 19642 ts_manager.cc:210] Registered new tserver with Master: 71498a460e3d4967965f341259b194f7 (127.18.102.1:45883)
I20261116 08:57:55.188840 18840 webserver.cc:577] Webserver started at http://127.18.102.2:40969/ using document root <none> and password file <none>
I20261116 08:57:55.189333 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:55.189522 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:55.189810 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:55.190315 19642 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36394
I20261116 08:57:55.190891 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-1-root/instance:
uuid: "86c41ad7b33348ac88342c984734507f"
format_stamp: "Formatted at 2026-11-16 08:57:55 on dist-test-slave-qt0z"
I20261116 08:57:55.195538 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20261116 08:57:55.199055 19781 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:55.199831 18840 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20261116 08:57:55.200105 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-1-root
uuid: "86c41ad7b33348ac88342c984734507f"
format_stamp: "Formatted at 2026-11-16 08:57:55 on dist-test-slave-qt0z"
I20261116 08:57:55.200384 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:55.222378 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:55.223449 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:55.224766 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:57:55.226775 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:57:55.226995 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:55.227252 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:57:55.227433 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:55.263077 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.2:40711
I20261116 08:57:55.263187 19843 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.2:40711 every 8 connection(s)
I20261116 08:57:55.268857 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:57:55.275790 19847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:57:55.277189 19848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:55.279227 18840 server_base.cc:1061] running on GCE node
I20261116 08:57:55.280819 19844 heartbeater.cc:345] Connected to a master server at 127.18.102.62:42427
W20261116 08:57:55.280862 19850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:57:55.281330 19844 heartbeater.cc:462] Registering TS with master...
I20261116 08:57:55.282027 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:57:55.282300 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:57:55.282224 19844 heartbeater.cc:508] Master 127.18.102.62:42427 requested a full tablet report, sending...
I20261116 08:57:55.282577 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819475282556 us; error 0 us; skew 500 ppm
I20261116 08:57:55.283373 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:57:55.284634 19642 ts_manager.cc:210] Registered new tserver with Master: 86c41ad7b33348ac88342c984734507f (127.18.102.2:40711)
I20261116 08:57:55.285851 18840 webserver.cc:577] Webserver started at http://127.18.102.3:38457/ using document root <none> and password file <none>
I20261116 08:57:55.286254 19642 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36402
I20261116 08:57:55.286338 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:57:55.286713 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:57:55.286984 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:57:55.288023 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-2-root/instance:
uuid: "434845759ef04f3488fa72004a4a7080"
format_stamp: "Formatted at 2026-11-16 08:57:55 on dist-test-slave-qt0z"
I20261116 08:57:55.292213 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20261116 08:57:55.295193 19855 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:55.296056 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261116 08:57:55.296286 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-2-root
uuid: "434845759ef04f3488fa72004a4a7080"
format_stamp: "Formatted at 2026-11-16 08:57:55 on dist-test-slave-qt0z"
I20261116 08:57:55.296517 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1794819469408847-18840-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:57:55.304677 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:57:55.305552 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:57:55.306788 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:57:55.308861 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:57:55.309027 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:55.309301 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:57:55.309437 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:57:55.369673 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.3:42013
I20261116 08:57:55.369755 19917 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.3:42013 every 8 connection(s)
I20261116 08:57:55.383769 19918 heartbeater.cc:345] Connected to a master server at 127.18.102.62:42427
I20261116 08:57:55.384209 19918 heartbeater.cc:462] Registering TS with master...
I20261116 08:57:55.385007 19918 heartbeater.cc:508] Master 127.18.102.62:42427 requested a full tablet report, sending...
I20261116 08:57:55.387228 19642 ts_manager.cc:210] Registered new tserver with Master: 434845759ef04f3488fa72004a4a7080 (127.18.102.3:42013)
I20261116 08:57:55.387460 18840 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013874528s
I20261116 08:57:55.388906 19642 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36408
I20261116 08:57:55.413559 19642 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20261116 08:57:55.414865 19642 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261116 08:57:55.454241 19734 tablet_service.cc:1500] Processing CreateTablet for tablet 4b4aec474a1e45369b37fcd7ef2717e6 (DEFAULT_TABLE table=test-table [id=e029789dc7ff4704be093e87f6b3a01d]), partition=
I20261116 08:57:55.455107 19809 tablet_service.cc:1500] Processing CreateTablet for tablet 4b4aec474a1e45369b37fcd7ef2717e6 (DEFAULT_TABLE table=test-table [id=e029789dc7ff4704be093e87f6b3a01d]), partition=
I20261116 08:57:55.455379 19734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4aec474a1e45369b37fcd7ef2717e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:55.456130 19809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4aec474a1e45369b37fcd7ef2717e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:55.456730 19883 tablet_service.cc:1500] Processing CreateTablet for tablet 4b4aec474a1e45369b37fcd7ef2717e6 (DEFAULT_TABLE table=test-table [id=e029789dc7ff4704be093e87f6b3a01d]), partition=
I20261116 08:57:55.457715 19883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4aec474a1e45369b37fcd7ef2717e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:57:55.472036 19933 tablet_bootstrap.cc:492] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7: Bootstrap starting.
I20261116 08:57:55.474691 19934 tablet_bootstrap.cc:492] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f: Bootstrap starting.
I20261116 08:57:55.476171 19935 tablet_bootstrap.cc:492] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080: Bootstrap starting.
I20261116 08:57:55.479017 19933 tablet_bootstrap.cc:654] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:55.482345 19934 tablet_bootstrap.cc:654] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:55.483385 19935 tablet_bootstrap.cc:654] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080: Neither blocks nor log segments found. Creating new log.
I20261116 08:57:55.484313 19933 tablet_bootstrap.cc:492] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7: No bootstrap required, opened a new log
I20261116 08:57:55.484704 19933 ts_tablet_manager.cc:1449] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20261116 08:57:55.488039 19933 raft_consensus.cc:360] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c41ad7b33348ac88342c984734507f" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 40711 } } peers { permanent_uuid: "71498a460e3d4967965f341259b194f7" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45883 } } peers { permanent_uuid: "434845759ef04f3488fa72004a4a7080" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 42013 } }
I20261116 08:57:55.488727 19933 raft_consensus.cc:386] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:57:55.489018 19933 raft_consensus.cc:748] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71498a460e3d4967965f341259b194f7, State: Initialized, Role: FOLLOWER
I20261116 08:57:55.489809 19933 consensus_queue.cc:261] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c41ad7b33348ac88342c984734507f" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 40711 } } peers { permanent_uuid: "71498a460e3d4967965f341259b194f7" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45883 } } peers { permanent_uuid: "434845759ef04f3488fa72004a4a7080" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 42013 } }
I20261116 08:57:55.489831 19934 tablet_bootstrap.cc:492] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f: No bootstrap required, opened a new log
I20261116 08:57:55.491503 19934 ts_tablet_manager.cc:1449] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20261116 08:57:55.492980 19935 tablet_bootstrap.cc:492] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080: No bootstrap required, opened a new log
I20261116 08:57:55.493182 19769 heartbeater.cc:500] Master 127.18.102.62:42427 was elected leader, sending a full tablet report...
I20261116 08:57:55.493415 19935 ts_tablet_manager.cc:1449] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20261116 08:57:55.493685 19933 ts_tablet_manager.cc:1480] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20261116 08:57:55.495718 19935 raft_consensus.cc:360] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c41ad7b33348ac88342c984734507f" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 40711 } } peers { permanent_uuid: "71498a460e3d4967965f341259b194f7" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45883 } } peers { permanent_uuid: "434845759ef04f3488fa72004a4a7080" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 42013 } }
I20261116 08:57:55.496274 19935 raft_consensus.cc:386] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:57:55.496565 19935 raft_consensus.cc:748] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 434845759ef04f3488fa72004a4a7080, State: Initialized, Role: FOLLOWER
I20261116 08:57:55.497193 19935 consensus_queue.cc:261] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c41ad7b33348ac88342c984734507f" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 40711 } } peers { permanent_uuid: "71498a460e3d4967965f341259b194f7" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45883 } } peers { permanent_uuid: "434845759ef04f3488fa72004a4a7080" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 42013 } }
I20261116 08:57:55.496091 19934 raft_consensus.cc:360] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c41ad7b33348ac88342c984734507f" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 40711 } } peers { permanent_uuid: "71498a460e3d4967965f341259b194f7" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45883 } } peers { permanent_uuid: "434845759ef04f3488fa72004a4a7080" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 42013 } }
I20261116 08:57:55.498301 19934 raft_consensus.cc:386] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:57:55.498796 19934 raft_consensus.cc:748] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86c41ad7b33348ac88342c984734507f, State: Initialized, Role: FOLLOWER
I20261116 08:57:55.500958 19934 consensus_queue.cc:261] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c41ad7b33348ac88342c984734507f" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 40711 } } peers { permanent_uuid: "71498a460e3d4967965f341259b194f7" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45883 } } peers { permanent_uuid: "434845759ef04f3488fa72004a4a7080" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 42013 } }
I20261116 08:57:55.505693 19844 heartbeater.cc:500] Master 127.18.102.62:42427 was elected leader, sending a full tablet report...
I20261116 08:57:55.506628 19935 ts_tablet_manager.cc:1480] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080: Time spent starting tablet: real 0.013s user 0.005s sys 0.005s
I20261116 08:57:55.507735 19934 ts_tablet_manager.cc:1480] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f: Time spent starting tablet: real 0.016s user 0.005s sys 0.002s
I20261116 08:57:55.508554 19918 heartbeater.cc:500] Master 127.18.102.62:42427 was elected leader, sending a full tablet report...
I20261116 08:57:55.535579 19940 raft_consensus.cc:499] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:57:55.535998 19940 raft_consensus.cc:521] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c41ad7b33348ac88342c984734507f" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 40711 } } peers { permanent_uuid: "71498a460e3d4967965f341259b194f7" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45883 } } peers { permanent_uuid: "434845759ef04f3488fa72004a4a7080" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 42013 } }
I20261116 08:57:55.537917 19940 leader_election.cc:290] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86c41ad7b33348ac88342c984734507f (127.18.102.2:40711), 71498a460e3d4967965f341259b194f7 (127.18.102.1:45883)
I20261116 08:57:55.546047 19819 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4b4aec474a1e45369b37fcd7ef2717e6" candidate_uuid: "434845759ef04f3488fa72004a4a7080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86c41ad7b33348ac88342c984734507f" is_pre_election: true
I20261116 08:57:55.546715 19819 raft_consensus.cc:2490] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 434845759ef04f3488fa72004a4a7080 in term 0.
I20261116 08:57:55.546862 19744 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4b4aec474a1e45369b37fcd7ef2717e6" candidate_uuid: "434845759ef04f3488fa72004a4a7080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71498a460e3d4967965f341259b194f7" is_pre_election: true
I20261116 08:57:55.547518 19744 raft_consensus.cc:2490] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 434845759ef04f3488fa72004a4a7080 in term 0.
I20261116 08:57:55.547902 19858 leader_election.cc:304] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 434845759ef04f3488fa72004a4a7080, 86c41ad7b33348ac88342c984734507f; no voters:
I20261116 08:57:55.548580 19940 raft_consensus.cc:2826] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 08:57:55.548821 19940 raft_consensus.cc:499] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 08:57:55.549036 19940 raft_consensus.cc:3096] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:55.553714 19940 raft_consensus.cc:521] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c41ad7b33348ac88342c984734507f" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 40711 } } peers { permanent_uuid: "71498a460e3d4967965f341259b194f7" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45883 } } peers { permanent_uuid: "434845759ef04f3488fa72004a4a7080" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 42013 } }
I20261116 08:57:55.555225 19940 leader_election.cc:290] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [CANDIDATE]: Term 1 election: Requested vote from peers 86c41ad7b33348ac88342c984734507f (127.18.102.2:40711), 71498a460e3d4967965f341259b194f7 (127.18.102.1:45883)
I20261116 08:57:55.555800 19819 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4b4aec474a1e45369b37fcd7ef2717e6" candidate_uuid: "434845759ef04f3488fa72004a4a7080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86c41ad7b33348ac88342c984734507f"
I20261116 08:57:55.556219 19819 raft_consensus.cc:3096] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:55.556147 19744 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4b4aec474a1e45369b37fcd7ef2717e6" candidate_uuid: "434845759ef04f3488fa72004a4a7080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71498a460e3d4967965f341259b194f7"
I20261116 08:57:55.556766 19744 raft_consensus.cc:3096] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:57:55.560585 19819 raft_consensus.cc:2490] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 434845759ef04f3488fa72004a4a7080 in term 1.
I20261116 08:57:55.560951 19744 raft_consensus.cc:2490] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 434845759ef04f3488fa72004a4a7080 in term 1.
I20261116 08:57:55.561471 19858 leader_election.cc:304] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 434845759ef04f3488fa72004a4a7080, 86c41ad7b33348ac88342c984734507f; no voters:
I20261116 08:57:55.562232 19940 raft_consensus.cc:2826] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:57:55.563012 19940 raft_consensus.cc:705] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 1 LEADER]: Becoming Leader. State: Replica: 434845759ef04f3488fa72004a4a7080, State: Running, Role: LEADER
I20261116 08:57:55.563822 19940 consensus_queue.cc:238] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c41ad7b33348ac88342c984734507f" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 40711 } } peers { permanent_uuid: "71498a460e3d4967965f341259b194f7" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45883 } } peers { permanent_uuid: "434845759ef04f3488fa72004a4a7080" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 42013 } }
I20261116 08:57:55.569350 19940 consensus_queue.cc:1049] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71498a460e3d4967965f341259b194f7" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 08:57:55.570365 19642 catalog_manager.cc:5840] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 reported cstate change: term changed from 0 to 1, leader changed from <none> to 434845759ef04f3488fa72004a4a7080 (127.18.102.3). New cstate: current_term: 1 leader_uuid: "434845759ef04f3488fa72004a4a7080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c41ad7b33348ac88342c984734507f" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71498a460e3d4967965f341259b194f7" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "434845759ef04f3488fa72004a4a7080" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 42013 } health_report { overall_health: HEALTHY } } }
I20261116 08:57:55.571080 19940 consensus_queue.cc:1049] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86c41ad7b33348ac88342c984734507f" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 08:58:00.640048 18840 tablet_replica.cc:333] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7: stopping tablet replica
I20261116 08:58:00.640717 18840 raft_consensus.cc:2261] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:00.641203 18840 raft_consensus.cc:2290] T 4b4aec474a1e45369b37fcd7ef2717e6 P 71498a460e3d4967965f341259b194f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:01.055895 19857 consensus_queue.cc:580] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [LEADER]: The tablet replica hosted on peer 71498a460e3d4967965f341259b194f7 has failed
W20261116 08:58:01.056371 19857 consensus_peers.cc:597] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 -> Peer 71498a460e3d4967965f341259b194f7 (127.18.102.1:45883): Couldn't send request to peer 71498a460e3d4967965f341259b194f7. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:01.165794 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
I20261116 08:58:01.191514 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:01.209296 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
I20261116 08:58:01.219364 18840 tablet_server.cc:179] TabletServer@127.18.102.2:0 shutting down...
I20261116 08:58:01.237300 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:01.237879 18840 tablet_replica.cc:333] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f: stopping tablet replica
I20261116 08:58:01.238339 18840 raft_consensus.cc:2261] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f [term 1 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:01.238713 18840 raft_consensus.cc:2290] T 4b4aec474a1e45369b37fcd7ef2717e6 P 86c41ad7b33348ac88342c984734507f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:01.256971 18840 tablet_server.cc:196] TabletServer@127.18.102.2:0 shutdown complete.
W20261116 08:58:01.265861 19858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.102.2:40711: connect: Connection refused (error 111)
I20261116 08:58:01.268836 18840 tablet_server.cc:179] TabletServer@127.18.102.3:0 shutting down...
W20261116 08:58:01.269999 19858 consensus_peers.cc:597] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 -> Peer 86c41ad7b33348ac88342c984734507f (127.18.102.2:40711): Couldn't send request to peer 86c41ad7b33348ac88342c984734507f. Status: Network error: Client connection negotiation failed: client connection to 127.18.102.2:40711: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:01.287038 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:01.287830 18840 tablet_replica.cc:333] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080: stopping tablet replica
I20261116 08:58:01.288368 18840 raft_consensus.cc:2261] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:01.289091 18840 raft_consensus.cc:2290] T 4b4aec474a1e45369b37fcd7ef2717e6 P 434845759ef04f3488fa72004a4a7080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:01.307274 18840 tablet_server.cc:196] TabletServer@127.18.102.3:0 shutdown complete.
I20261116 08:58:01.316105 18840 master.cc:585] Master@127.18.102.62:42427 shutting down...
I20261116 08:58:01.334714 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:01.335413 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:01.335759 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P 76b58a7bde4a4164a118eb3b9592b62a: stopping tablet replica
I20261116 08:58:01.353368 18840 master.cc:607] Master@127.18.102.62:42427 shutdown complete.
[ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (6500 ms)
[ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20261116 08:58:01.385888 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:38959
I20261116 08:58:01.386971 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:01.391968 19962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:01.393447 19963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:01.394825 19965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:01.395040 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:01.395790 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:01.395975 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:01.396090 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819481396079 us; error 0 us; skew 500 ppm
I20261116 08:58:01.396502 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:01.398524 18840 webserver.cc:577] Webserver started at http://127.18.102.62:44191/ using document root <none> and password file <none>
I20261116 08:58:01.398933 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:01.399072 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:01.399349 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:01.400345 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "2296866b54264286bdaca70da09bf2c0"
format_stamp: "Formatted at 2026-11-16 08:58:01 on dist-test-slave-qt0z"
I20261116 08:58:01.404620 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20261116 08:58:01.407624 19970 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:01.408320 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261116 08:58:01.408541 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "2296866b54264286bdaca70da09bf2c0"
format_stamp: "Formatted at 2026-11-16 08:58:01 on dist-test-slave-qt0z"
I20261116 08:58:01.408768 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:01.433985 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:01.435089 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:01.470774 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:38959
I20261116 08:58:01.470867 20021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:38959 every 8 connection(s)
I20261116 08:58:01.474749 20022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:01.485070 20022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0: Bootstrap starting.
I20261116 08:58:01.489984 20022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:01.493999 20022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0: No bootstrap required, opened a new log
I20261116 08:58:01.496119 20022 raft_consensus.cc:360] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2296866b54264286bdaca70da09bf2c0" member_type: VOTER }
I20261116 08:58:01.496511 20022 raft_consensus.cc:386] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:01.496701 20022 raft_consensus.cc:748] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2296866b54264286bdaca70da09bf2c0, State: Initialized, Role: FOLLOWER
I20261116 08:58:01.497172 20022 consensus_queue.cc:261] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2296866b54264286bdaca70da09bf2c0" member_type: VOTER }
I20261116 08:58:01.497576 20022 raft_consensus.cc:400] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:01.497752 20022 raft_consensus.cc:499] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:01.497949 20022 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:01.501796 20022 raft_consensus.cc:521] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2296866b54264286bdaca70da09bf2c0" member_type: VOTER }
I20261116 08:58:01.502233 20022 leader_election.cc:304] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2296866b54264286bdaca70da09bf2c0; no voters:
I20261116 08:58:01.503304 20022 leader_election.cc:290] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:01.503654 20025 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:01.504868 20025 raft_consensus.cc:705] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [term 1 LEADER]: Becoming Leader. State: Replica: 2296866b54264286bdaca70da09bf2c0, State: Running, Role: LEADER
I20261116 08:58:01.505518 20025 consensus_queue.cc:238] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2296866b54264286bdaca70da09bf2c0" member_type: VOTER }
I20261116 08:58:01.506193 20022 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:01.508173 20026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2296866b54264286bdaca70da09bf2c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2296866b54264286bdaca70da09bf2c0" member_type: VOTER } }
I20261116 08:58:01.508173 20027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2296866b54264286bdaca70da09bf2c0. Latest consensus state: current_term: 1 leader_uuid: "2296866b54264286bdaca70da09bf2c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2296866b54264286bdaca70da09bf2c0" member_type: VOTER } }
I20261116 08:58:01.508769 20026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:01.508883 20027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:01.514933 20030 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:58:01.522409 20030 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:58:01.523284 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:58:01.530774 20030 catalog_manager.cc:1398] Generated new cluster ID: 8eccfa141af34bae80af19c6af0cc13d
I20261116 08:58:01.531035 20030 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:58:01.559990 20030 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:58:01.561209 20030 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:58:01.576756 20030 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0: Generated new TSK 0
I20261116 08:58:01.577322 20030 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:58:01.589190 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:01.594776 20043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:01.596081 20044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:01.597434 20046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:01.597644 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:01.598467 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:01.598671 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:01.598824 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819481598811 us; error 0 us; skew 500 ppm
I20261116 08:58:01.599349 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:01.601378 18840 webserver.cc:577] Webserver started at http://127.18.102.1:42135/ using document root <none> and password file <none>
I20261116 08:58:01.601814 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:01.601963 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:01.602162 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:01.603118 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "c052b436e60f48f4b790ec547cd8f4bf"
format_stamp: "Formatted at 2026-11-16 08:58:01 on dist-test-slave-qt0z"
I20261116 08:58:01.607468 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20261116 08:58:01.610467 20051 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:01.611246 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261116 08:58:01.611513 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "c052b436e60f48f4b790ec547cd8f4bf"
format_stamp: "Formatted at 2026-11-16 08:58:01 on dist-test-slave-qt0z"
I20261116 08:58:01.611793 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:01.623620 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:01.624722 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:01.626046 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:01.628115 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:01.628325 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:01.628566 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:01.628736 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:01.666268 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:39165
I20261116 08:58:01.666365 20113 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:39165 every 8 connection(s)
I20261116 08:58:01.680356 20114 heartbeater.cc:345] Connected to a master server at 127.18.102.62:38959
I20261116 08:58:01.680701 20114 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:01.681380 20114 heartbeater.cc:508] Master 127.18.102.62:38959 requested a full tablet report, sending...
I20261116 08:58:01.683362 19987 ts_manager.cc:210] Registered new tserver with Master: c052b436e60f48f4b790ec547cd8f4bf (127.18.102.1:39165)
I20261116 08:58:01.683480 18840 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.014005492s
I20261116 08:58:01.685146 19987 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45738
I20261116 08:58:01.702454 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
I20261116 08:58:01.717012 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:01.731922 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
I20261116 08:58:01.738957 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261116 08:58:01.741533 18840 tablet_server.cc:94] deduplicated master addresses: 127.18.102.62:38959
W20261116 08:58:01.744740 20124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:01.744764 20125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:01.745968 20127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:01.746831 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:01.747663 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:01.747848 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:01.748008 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819481747977 us; error 0 us; skew 500 ppm
I20261116 08:58:01.748549 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:01.750551 18840 webserver.cc:577] Webserver started at http://127.18.102.1:42135/ using document root <none> and password file <none>
I20261116 08:58:01.751003 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:01.751197 18840 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261116 08:58:01.754432 18840 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20261116 08:58:01.756799 20132 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:01.757483 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261116 08:58:01.757750 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "c052b436e60f48f4b790ec547cd8f4bf"
format_stamp: "Formatted at 2026-11-16 08:58:01 on dist-test-slave-qt0z"
I20261116 08:58:01.758026 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:01.795045 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:01.796057 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:01.797364 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:01.799549 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:01.799798 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:01.800017 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:01.800179 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:01.835074 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:39165
I20261116 08:58:01.835227 20194 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:39165 every 8 connection(s)
I20261116 08:58:01.839957 18840 tablet_server.cc:179] TabletServer@127.18.102.1:39165 shutting down...
I20261116 08:58:01.861459 20195 heartbeater.cc:345] Connected to a master server at 127.18.102.62:38959
I20261116 08:58:01.861908 20195 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:01.862565 20195 heartbeater.cc:508] Master 127.18.102.62:38959 requested a full tablet report, sending...
I20261116 08:58:01.864463 19987 ts_manager.cc:210] Re-registered known tserver with Master: c052b436e60f48f4b790ec547cd8f4bf (127.18.102.1:39165)
I20261116 08:58:01.865659 19987 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45754
I20261116 08:58:01.867867 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:01.882392 18840 tablet_server.cc:196] TabletServer@127.18.102.1:39165 shutdown complete.
I20261116 08:58:01.888840 18840 master.cc:585] Master@127.18.102.62:38959 shutting down...
I20261116 08:58:01.902374 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:01.902851 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:01.903319 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2296866b54264286bdaca70da09bf2c0: stopping tablet replica
I20261116 08:58:01.921866 18840 master.cc:607] Master@127.18.102.62:38959 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (556 ms)
[ RUN ] TsTabletManagerITest.TestTableStats
I20261116 08:58:01.942895 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:32983,127.18.102.61:36365,127.18.102.60:32843
I20261116 08:58:01.944177 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:01.948622 20202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:01.949851 20203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:01.951040 20205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:01.952135 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:01.952916 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:01.953125 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:01.953299 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819481953267 us; error 0 us; skew 500 ppm
I20261116 08:58:01.953831 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:01.955988 18840 webserver.cc:577] Webserver started at http://127.18.102.62:40393/ using document root <none> and password file <none>
I20261116 08:58:01.956437 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:01.956620 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:01.956882 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:01.957895 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "d9a31f175a1a4361aace9c508bdb5291"
format_stamp: "Formatted at 2026-11-16 08:58:01 on dist-test-slave-qt0z"
I20261116 08:58:01.962092 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20261116 08:58:01.965121 20210 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:01.965823 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20261116 08:58:01.966096 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "d9a31f175a1a4361aace9c508bdb5291"
format_stamp: "Formatted at 2026-11-16 08:58:01 on dist-test-slave-qt0z"
I20261116 08:58:01.966379 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:01.983285 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:01.984225 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:02.018673 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:32983
I20261116 08:58:02.018764 20261 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:32983 every 8 connection(s)
I20261116 08:58:02.022454 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261116 08:58:02.022610 20262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20261116 08:58:02.027689 20264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:02.029623 20265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.030566 20262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } has no permanent_uuid. Determining permanent_uuid...
I20261116 08:58:02.030823 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:02.031852 20267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.032894 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:02.033175 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:02.033377 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819482033359 us; error 0 us; skew 500 ppm
I20261116 08:58:02.034180 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:02.037161 18840 webserver.cc:577] Webserver started at http://127.18.102.61:40741/ using document root <none> and password file <none>
I20261116 08:58:02.037632 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:02.037832 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:02.038105 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:02.039513 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-1-root/instance:
uuid: "9188aed91ff24b0ea766bf89f306b495"
format_stamp: "Formatted at 2026-11-16 08:58:02 on dist-test-slave-qt0z"
I20261116 08:58:02.043931 20262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } has no permanent_uuid. Determining permanent_uuid...
I20261116 08:58:02.045361 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
W20261116 08:58:02.049206 20262 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.102.61:36365: Network error: Client connection negotiation failed: client connection to 127.18.102.61:36365: connect: Connection refused (error 111)
I20261116 08:58:02.049659 20275 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:02.050369 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261116 08:58:02.050643 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-1-root
uuid: "9188aed91ff24b0ea766bf89f306b495"
format_stamp: "Formatted at 2026-11-16 08:58:02 on dist-test-slave-qt0z"
I20261116 08:58:02.050889 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:02.060262 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:02.061154 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:02.093922 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.61:36365
I20261116 08:58:02.094019 20326 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.61:36365 every 8 connection(s)
I20261116 08:58:02.097407 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261116 08:58:02.097615 20327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:02.100672 20262 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } attempt: 1
W20261116 08:58:02.103282 20329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:02.103431 20330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.106860 20327 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } has no permanent_uuid. Determining permanent_uuid...
W20261116 08:58:02.117633 20333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.117987 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:02.118599 20262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } has no permanent_uuid. Determining permanent_uuid...
I20261116 08:58:02.118814 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:02.119095 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:02.119344 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819482119324 us; error 0 us; skew 500 ppm
I20261116 08:58:02.120059 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20261116 08:58:02.123844 20262 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.102.60:32843: Network error: Client connection negotiation failed: client connection to 127.18.102.60:32843: connect: Connection refused (error 111)
I20261116 08:58:02.124446 18840 webserver.cc:577] Webserver started at http://127.18.102.60:39727/ using document root <none> and password file <none>
I20261116 08:58:02.124657 20327 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } has no permanent_uuid. Determining permanent_uuid...
I20261116 08:58:02.125105 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:02.125360 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:02.125681 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:02.127111 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-2-root/instance:
uuid: "a356758e034e49e292899b82f51810cd"
format_stamp: "Formatted at 2026-11-16 08:58:02 on dist-test-slave-qt0z"
I20261116 08:58:02.132663 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20261116 08:58:02.134356 20327 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } has no permanent_uuid. Determining permanent_uuid...
I20261116 08:58:02.136694 20339 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:02.137603 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261116 08:58:02.137866 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-2-root
uuid: "a356758e034e49e292899b82f51810cd"
format_stamp: "Formatted at 2026-11-16 08:58:02 on dist-test-slave-qt0z"
I20261116 08:58:02.138183 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20261116 08:58:02.139335 20327 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.102.60:32843: Network error: Client connection negotiation failed: client connection to 127.18.102.60:32843: connect: Connection refused (error 111)
I20261116 08:58:02.157732 20327 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } attempt: 1
I20261116 08:58:02.161094 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20261116 08:58:02.161777 20327 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.102.60:32843: Network error: Client connection negotiation failed: client connection to 127.18.102.60:32843: connect: Connection refused (error 111)
I20261116 08:58:02.162374 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:02.169289 20262 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } attempt: 1
W20261116 08:58:02.173758 20262 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.102.60:32843: Network error: Client connection negotiation failed: client connection to 127.18.102.60:32843: connect: Connection refused (error 111)
I20261116 08:58:02.200608 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.60:32843
I20261116 08:58:02.200683 20391 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.60:32843 every 8 connection(s)
I20261116 08:58:02.203578 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:58:02.204272 20392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:02.209918 20392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } has no permanent_uuid. Determining permanent_uuid...
I20261116 08:58:02.214244 20262 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } attempt: 2
I20261116 08:58:02.216157 20327 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } attempt: 2
I20261116 08:58:02.223383 20392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } has no permanent_uuid. Determining permanent_uuid...
I20261116 08:58:02.234660 20262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291: Bootstrap starting.
I20261116 08:58:02.235580 20392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } has no permanent_uuid. Determining permanent_uuid...
I20261116 08:58:02.235966 20327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: Bootstrap starting.
I20261116 08:58:02.240523 20262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:02.241245 20327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:02.245743 20262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291: No bootstrap required, opened a new log
I20261116 08:58:02.245798 20327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: No bootstrap required, opened a new log
I20261116 08:58:02.248251 20262 raft_consensus.cc:360] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:02.248958 20262 raft_consensus.cc:386] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:02.248593 20327 raft_consensus.cc:360] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:02.249294 20262 raft_consensus.cc:748] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a31f175a1a4361aace9c508bdb5291, State: Initialized, Role: FOLLOWER
I20261116 08:58:02.249450 20327 raft_consensus.cc:386] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:02.249956 20327 raft_consensus.cc:748] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9188aed91ff24b0ea766bf89f306b495, State: Initialized, Role: FOLLOWER
I20261116 08:58:02.250038 20262 consensus_queue.cc:261] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:02.250633 20327 consensus_queue.cc:261] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:02.252707 20400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:02.253470 20400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:02.254262 20327 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:02.255909 20399 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:02.256606 20399 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:02.257408 20262 sys_catalog.cc:565] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:02.261850 20392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: Bootstrap starting.
I20261116 08:58:02.270457 20399 raft_consensus.cc:499] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:02.271097 20392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:02.270881 20399 raft_consensus.cc:521] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
W20261116 08:58:02.271528 20412 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20261116 08:58:02.271878 20412 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20261116 08:58:02.273308 20399 leader_election.cc:290] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9188aed91ff24b0ea766bf89f306b495 (127.18.102.61:36365), a356758e034e49e292899b82f51810cd (127.18.102.60:32843)
I20261116 08:58:02.274082 20302 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9a31f175a1a4361aace9c508bdb5291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9188aed91ff24b0ea766bf89f306b495" is_pre_election: true
I20261116 08:58:02.275002 20302 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9a31f175a1a4361aace9c508bdb5291 in term 0.
I20261116 08:58:02.276592 20214 leader_election.cc:304] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9188aed91ff24b0ea766bf89f306b495, d9a31f175a1a4361aace9c508bdb5291; no voters:
I20261116 08:58:02.277369 20399 raft_consensus.cc:2826] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 08:58:02.277770 20399 raft_consensus.cc:499] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 08:58:02.278272 20399 raft_consensus.cc:3096] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:02.280542 20392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: No bootstrap required, opened a new log
I20261116 08:58:02.284188 20392 raft_consensus.cc:360] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:02.285060 20392 raft_consensus.cc:386] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:02.285400 20392 raft_consensus.cc:748] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a356758e034e49e292899b82f51810cd, State: Initialized, Role: FOLLOWER
I20261116 08:58:02.286237 20399 raft_consensus.cc:521] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:02.286324 20392 consensus_queue.cc:261] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:02.289144 20302 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9a31f175a1a4361aace9c508bdb5291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9188aed91ff24b0ea766bf89f306b495"
I20261116 08:58:02.289705 20302 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:02.290824 20399 leader_election.cc:290] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 1 election: Requested vote from peers 9188aed91ff24b0ea766bf89f306b495 (127.18.102.61:36365), a356758e034e49e292899b82f51810cd (127.18.102.60:32843)
I20261116 08:58:02.292672 20416 sys_catalog.cc:455] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:02.293618 20416 sys_catalog.cc:458] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:02.296578 20392 sys_catalog.cc:565] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:02.296763 20366 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9a31f175a1a4361aace9c508bdb5291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a356758e034e49e292899b82f51810cd"
I20261116 08:58:02.297389 20366 raft_consensus.cc:3096] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:02.297605 20302 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a31f175a1a4361aace9c508bdb5291 in term 1.
I20261116 08:58:02.298854 20214 leader_election.cc:304] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9188aed91ff24b0ea766bf89f306b495, d9a31f175a1a4361aace9c508bdb5291; no voters:
I20261116 08:58:02.303390 20367 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9a31f175a1a4361aace9c508bdb5291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a356758e034e49e292899b82f51810cd" is_pre_election: true
I20261116 08:58:02.305096 20399 raft_consensus.cc:2826] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:02.311617 20366 raft_consensus.cc:2490] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a31f175a1a4361aace9c508bdb5291 in term 1.
I20261116 08:58:02.316617 20399 raft_consensus.cc:705] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 1 LEADER]: Becoming Leader. State: Replica: d9a31f175a1a4361aace9c508bdb5291, State: Running, Role: LEADER
I20261116 08:58:02.317556 20399 consensus_queue.cc:238] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:02.322455 20421 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a31f175a1a4361aace9c508bdb5291. Latest consensus state: current_term: 1 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:02.323236 20421 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:02.337225 20438 catalog_manager.cc:820] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20261116 08:58:02.339210 20421 consensus_queue.cc:1049] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261116 08:58:02.339702 20431 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:58:02.340921 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20261116 08:58:02.341192 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20261116 08:58:02.342556 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261116 08:58:02.340385 20432 consensus_queue.cc:1049] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
E20261116 08:58:02.344653 18840 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20261116 08:58:02.346618 20431 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:58:02.353472 20416 sys_catalog.cc:455] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a31f175a1a4361aace9c508bdb5291. Latest consensus state: current_term: 1 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:02.354027 20416 sys_catalog.cc:458] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:02.356892 20400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a31f175a1a4361aace9c508bdb5291. Latest consensus state: current_term: 1 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:02.357911 20400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:02.360980 20399 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:02.361953 20399 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:02.365091 20432 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:02.366181 20432 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: This master's current role is: LEADER
W20261116 08:58:02.370180 20440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.373668 20400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:02.374394 20400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:02.374517 20416 sys_catalog.cc:455] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:02.375473 20416 sys_catalog.cc:458] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: This master's current role is: FOLLOWER
W20261116 08:58:02.380424 20444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.390738 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:02.392103 20448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.393682 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:02.393957 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:02.394165 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819482394145 us; error 0 us; skew 500 ppm
I20261116 08:58:02.394843 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:02.395471 20431 catalog_manager.cc:1398] Generated new cluster ID: f1b7bc8047cd4836a24134372a439417
I20261116 08:58:02.395809 20431 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:58:02.398929 18840 webserver.cc:577] Webserver started at http://127.18.102.1:36713/ using document root <none> and password file <none>
I20261116 08:58:02.399533 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:02.399760 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:02.400074 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:02.401367 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "d92186d919b741b4b8d69416caad3021"
format_stamp: "Formatted at 2026-11-16 08:58:02 on dist-test-slave-qt0z"
I20261116 08:58:02.406175 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20261116 08:58:02.409708 20456 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:02.410511 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261116 08:58:02.410791 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "d92186d919b741b4b8d69416caad3021"
format_stamp: "Formatted at 2026-11-16 08:58:02 on dist-test-slave-qt0z"
I20261116 08:58:02.411082 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:02.431766 20431 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:58:02.433295 20431 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:58:02.460731 20431 catalog_manager.cc:6221] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291: Generated new TSK 0
I20261116 08:58:02.461333 20431 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:58:02.462297 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:02.463910 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:02.465631 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:02.467813 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:02.467993 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:02.468266 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:02.468443 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:02.509840 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:37237
I20261116 08:58:02.509922 20519 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:37237 every 8 connection(s)
I20261116 08:58:02.525050 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261116 08:58:02.528489 18840 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20261116 08:58:02.535935 20520 heartbeater.cc:345] Connected to a master server at 127.18.102.60:32843
I20261116 08:58:02.536394 20520 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:02.537437 20520 heartbeater.cc:508] Master 127.18.102.60:32843 requested a full tablet report, sending...
I20261116 08:58:02.541445 20357 ts_manager.cc:210] Registered new tserver with Master: d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237)
W20261116 08:58:02.542449 20528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.543675 20521 heartbeater.cc:345] Connected to a master server at 127.18.102.61:36365
I20261116 08:58:02.544134 20521 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:02.545258 20521 heartbeater.cc:508] Master 127.18.102.61:36365 requested a full tablet report, sending...
W20261116 08:58:02.549150 20527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.550402 20292 ts_manager.cc:210] Registered new tserver with Master: d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237)
I20261116 08:58:02.551316 20522 heartbeater.cc:345] Connected to a master server at 127.18.102.62:32983
I20261116 08:58:02.551826 20522 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:02.552982 20522 heartbeater.cc:508] Master 127.18.102.62:32983 requested a full tablet report, sending...
I20261116 08:58:02.556212 20227 ts_manager.cc:210] Registered new tserver with Master: d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237)
I20261116 08:58:02.559255 20227 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55660
I20261116 08:58:02.559837 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:02.560248 20530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.561241 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:02.561487 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:02.561753 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819482561733 us; error 0 us; skew 500 ppm
I20261116 08:58:02.562512 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:02.564719 18840 webserver.cc:577] Webserver started at http://127.18.102.2:44279/ using document root <none> and password file <none>
I20261116 08:58:02.565193 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:02.565407 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:02.565672 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:02.566680 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-1-root/instance:
uuid: "e950e37c4d2d425a9771e063fdaa2850"
format_stamp: "Formatted at 2026-11-16 08:58:02 on dist-test-slave-qt0z"
I20261116 08:58:02.570834 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20261116 08:58:02.574265 20535 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:02.574983 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261116 08:58:02.575286 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-1-root
uuid: "e950e37c4d2d425a9771e063fdaa2850"
format_stamp: "Formatted at 2026-11-16 08:58:02 on dist-test-slave-qt0z"
I20261116 08:58:02.575579 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:02.599995 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:02.601168 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:02.602515 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:02.605232 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:02.605454 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:02.605690 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:02.605859 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:02.661959 20522 heartbeater.cc:500] Master 127.18.102.62:32983 was elected leader, sending a full tablet report...
I20261116 08:58:02.681226 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.2:37591
I20261116 08:58:02.681314 20597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.2:37591 every 8 connection(s)
I20261116 08:58:02.696478 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261116 08:58:02.698877 18840 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20261116 08:58:02.724778 20599 heartbeater.cc:345] Connected to a master server at 127.18.102.61:36365
W20261116 08:58:02.724779 20606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.725558 20599 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:02.726843 20599 heartbeater.cc:508] Master 127.18.102.61:36365 requested a full tablet report, sending...
I20261116 08:58:02.728089 20598 heartbeater.cc:345] Connected to a master server at 127.18.102.60:32843
I20261116 08:58:02.728528 20598 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:02.729282 20598 heartbeater.cc:508] Master 127.18.102.60:32843 requested a full tablet report, sending...
W20261116 08:58:02.729609 20605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.730237 20292 ts_manager.cc:210] Registered new tserver with Master: e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:02.732070 20357 ts_manager.cc:210] Registered new tserver with Master: e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:02.734091 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:02.734038 20608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:02.735441 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:02.735666 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:02.735915 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819482735897 us; error 0 us; skew 500 ppm
I20261116 08:58:02.736467 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:02.738234 20600 heartbeater.cc:345] Connected to a master server at 127.18.102.62:32983
I20261116 08:58:02.738576 20600 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:02.739248 18840 webserver.cc:577] Webserver started at http://127.18.102.3:36559/ using document root <none> and password file <none>
I20261116 08:58:02.739328 20600 heartbeater.cc:508] Master 127.18.102.62:32983 requested a full tablet report, sending...
I20261116 08:58:02.739805 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:02.740046 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:02.740406 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:02.741696 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-2-root/instance:
uuid: "f5439f672b0b4560aa83c325f85cc6aa"
format_stamp: "Formatted at 2026-11-16 08:58:02 on dist-test-slave-qt0z"
I20261116 08:58:02.741719 20227 ts_manager.cc:210] Registered new tserver with Master: e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:02.743721 20227 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55670
I20261116 08:58:02.747252 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20261116 08:58:02.750978 20613 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:02.752027 18840 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20261116 08:58:02.752339 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-2-root
uuid: "f5439f672b0b4560aa83c325f85cc6aa"
format_stamp: "Formatted at 2026-11-16 08:58:02 on dist-test-slave-qt0z"
I20261116 08:58:02.752631 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:02.773699 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:02.775027 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:02.776371 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:02.778471 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:02.778705 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:02.778990 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:02.779201 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20261116 08:58:02.846144 20600 heartbeater.cc:500] Master 127.18.102.62:32983 was elected leader, sending a full tablet report...
I20261116 08:58:02.852200 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.3:35323
I20261116 08:58:02.852448 20675 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.3:35323 every 8 connection(s)
I20261116 08:58:02.879230 20676 heartbeater.cc:345] Connected to a master server at 127.18.102.60:32843
I20261116 08:58:02.879688 20676 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:02.880621 20676 heartbeater.cc:508] Master 127.18.102.60:32843 requested a full tablet report, sending...
I20261116 08:58:02.882166 20679 heartbeater.cc:345] Connected to a master server at 127.18.102.62:32983
I20261116 08:58:02.882839 20679 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:02.883684 20357 ts_manager.cc:210] Registered new tserver with Master: f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:02.883806 20679 heartbeater.cc:508] Master 127.18.102.62:32983 requested a full tablet report, sending...
I20261116 08:58:02.886073 20677 heartbeater.cc:345] Connected to a master server at 127.18.102.61:36365
I20261116 08:58:02.886224 20227 ts_manager.cc:210] Registered new tserver with Master: f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:02.886468 20677 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:02.887403 20677 heartbeater.cc:508] Master 127.18.102.61:36365 requested a full tablet report, sending...
I20261116 08:58:02.888039 20227 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55676
I20261116 08:58:02.889950 20292 ts_manager.cc:210] Registered new tserver with Master: f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:02.890074 18840 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016801452s
I20261116 08:58:02.921250 20227 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20261116 08:58:02.922931 20227 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261116 08:58:02.990648 20679 heartbeater.cc:500] Master 127.18.102.62:32983 was elected leader, sending a full tablet report...
I20261116 08:58:03.005369 20563 tablet_service.cc:1500] Processing CreateTablet for tablet 19ac1f3e0f6c4b37b015139ac43a746a (DEFAULT_TABLE table=test-table [id=57a542ce297949569849767182f5b9d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:03.006102 20562 tablet_service.cc:1500] Processing CreateTablet for tablet 86badd1d1e29431daebce9a9ae80f0c9 (DEFAULT_TABLE table=test-table [id=57a542ce297949569849767182f5b9d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:03.006943 20563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19ac1f3e0f6c4b37b015139ac43a746a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:03.008165 20561 tablet_service.cc:1500] Processing CreateTablet for tablet 8663ba9cebca4dd0922589c3095d07f1 (DEFAULT_TABLE table=test-table [id=57a542ce297949569849767182f5b9d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:03.009615 20561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8663ba9cebca4dd0922589c3095d07f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:03.010836 20485 tablet_service.cc:1500] Processing CreateTablet for tablet 19ac1f3e0f6c4b37b015139ac43a746a (DEFAULT_TABLE table=test-table [id=57a542ce297949569849767182f5b9d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:03.012187 20485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19ac1f3e0f6c4b37b015139ac43a746a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:03.013929 20641 tablet_service.cc:1500] Processing CreateTablet for tablet 19ac1f3e0f6c4b37b015139ac43a746a (DEFAULT_TABLE table=test-table [id=57a542ce297949569849767182f5b9d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:03.014469 20640 tablet_service.cc:1500] Processing CreateTablet for tablet 86badd1d1e29431daebce9a9ae80f0c9 (DEFAULT_TABLE table=test-table [id=57a542ce297949569849767182f5b9d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:03.015431 20641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19ac1f3e0f6c4b37b015139ac43a746a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:03.016145 20640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86badd1d1e29431daebce9a9ae80f0c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:03.016531 20562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86badd1d1e29431daebce9a9ae80f0c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:03.016207 20484 tablet_service.cc:1500] Processing CreateTablet for tablet 86badd1d1e29431daebce9a9ae80f0c9 (DEFAULT_TABLE table=test-table [id=57a542ce297949569849767182f5b9d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:03.019091 20483 tablet_service.cc:1500] Processing CreateTablet for tablet 8663ba9cebca4dd0922589c3095d07f1 (DEFAULT_TABLE table=test-table [id=57a542ce297949569849767182f5b9d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:03.020023 20639 tablet_service.cc:1500] Processing CreateTablet for tablet 8663ba9cebca4dd0922589c3095d07f1 (DEFAULT_TABLE table=test-table [id=57a542ce297949569849767182f5b9d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:03.022040 20639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8663ba9cebca4dd0922589c3095d07f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:03.023423 20484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86badd1d1e29431daebce9a9ae80f0c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:03.028131 20483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8663ba9cebca4dd0922589c3095d07f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:03.071388 20697 tablet_bootstrap.cc:492] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850: Bootstrap starting.
I20261116 08:58:03.076853 20698 tablet_bootstrap.cc:492] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021: Bootstrap starting.
I20261116 08:58:03.086835 20699 tablet_bootstrap.cc:492] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa: Bootstrap starting.
I20261116 08:58:03.089637 20697 tablet_bootstrap.cc:654] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:03.092360 20698 tablet_bootstrap.cc:654] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:03.092825 20699 tablet_bootstrap.cc:654] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:03.101686 20699 tablet_bootstrap.cc:492] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa: No bootstrap required, opened a new log
I20261116 08:58:03.101830 20697 tablet_bootstrap.cc:492] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850: No bootstrap required, opened a new log
I20261116 08:58:03.102310 20699 ts_tablet_manager.cc:1449] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.002s
I20261116 08:58:03.102468 20697 ts_tablet_manager.cc:1449] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.007s
I20261116 08:58:03.104296 20698 tablet_bootstrap.cc:492] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021: No bootstrap required, opened a new log
I20261116 08:58:03.104748 20698 ts_tablet_manager.cc:1449] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.011s
I20261116 08:58:03.106416 20697 raft_consensus.cc:360] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.107092 20697 raft_consensus.cc:386] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:03.107394 20697 raft_consensus.cc:748] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Initialized, Role: FOLLOWER
I20261116 08:58:03.108100 20697 consensus_queue.cc:261] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.110390 20699 raft_consensus.cc:360] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.111058 20699 raft_consensus.cc:386] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:03.111343 20698 raft_consensus.cc:360] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.112094 20698 raft_consensus.cc:386] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:03.112378 20698 raft_consensus.cc:748] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92186d919b741b4b8d69416caad3021, State: Initialized, Role: FOLLOWER
I20261116 08:58:03.112624 20697 ts_tablet_manager.cc:1480] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20261116 08:58:03.112983 20698 consensus_queue.cc:261] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.113653 20697 tablet_bootstrap.cc:492] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850: Bootstrap starting.
I20261116 08:58:03.117209 20698 ts_tablet_manager.cc:1480] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021: Time spent starting tablet: real 0.012s user 0.003s sys 0.005s
I20261116 08:58:03.119554 20698 tablet_bootstrap.cc:492] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021: Bootstrap starting.
I20261116 08:58:03.120781 20697 tablet_bootstrap.cc:654] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:03.111373 20699 raft_consensus.cc:748] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5439f672b0b4560aa83c325f85cc6aa, State: Initialized, Role: FOLLOWER
I20261116 08:58:03.121672 20703 raft_consensus.cc:499] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:03.122556 20703 raft_consensus.cc:521] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.124269 20699 consensus_queue.cc:261] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.127362 20703 leader_election.cc:290] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:03.138285 20698 tablet_bootstrap.cc:654] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:03.148921 20495 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:03.149684 20495 raft_consensus.cc:2490] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 0.
I20261116 08:58:03.159773 20536 leader_election.cc:304] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850; no voters:
I20261116 08:58:03.161741 20703 raft_consensus.cc:2826] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 08:58:03.162128 20703 raft_consensus.cc:499] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 08:58:03.162489 20703 raft_consensus.cc:3096] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:03.175551 20706 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:03.185505 20699 ts_tablet_manager.cc:1480] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa: Time spent starting tablet: real 0.083s user 0.015s sys 0.005s
I20261116 08:58:03.185417 20706 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.187494 20699 tablet_bootstrap.cc:492] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa: Bootstrap starting.
I20261116 08:58:03.187317 20651 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
W20261116 08:58:03.190204 20538 leader_election.cc:343] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323): Illegal state: must be running to vote when last-logged opid is not known
I20261116 08:58:03.205317 20703 raft_consensus.cc:521] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.207739 20703 leader_election.cc:290] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 1 election: Requested vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:03.208238 20495 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021"
I20261116 08:58:03.208760 20495 raft_consensus.cc:3096] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:03.192323 20706 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:03.210695 20651 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
W20261116 08:58:03.213197 20538 leader_election.cc:343] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323): Illegal state: must be running to vote when last-logged opid is not known
I20261116 08:58:03.214444 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:03.216895 20573 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
W20261116 08:58:03.217710 20614 leader_election.cc:343] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237): Illegal state: must be running to vote when last-logged opid is not known
W20261116 08:58:03.219480 20616 leader_election.cc:343] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591): Illegal state: must be running to vote when last-logged opid is not known
I20261116 08:58:03.219884 20697 tablet_bootstrap.cc:492] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850: No bootstrap required, opened a new log
I20261116 08:58:03.219947 20616 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5439f672b0b4560aa83c325f85cc6aa; no voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850
I20261116 08:58:03.220919 20697 ts_tablet_manager.cc:1449] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850: Time spent bootstrapping tablet: real 0.107s user 0.009s sys 0.013s
I20261116 08:58:03.220968 20495 raft_consensus.cc:2490] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 1.
I20261116 08:58:03.221306 20706 raft_consensus.cc:2771] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261116 08:58:03.220536 20698 tablet_bootstrap.cc:492] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021: No bootstrap required, opened a new log
I20261116 08:58:03.222249 20536 leader_election.cc:304] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850; no voters: f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:03.222894 20698 ts_tablet_manager.cc:1449] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021: Time spent bootstrapping tablet: real 0.104s user 0.008s sys 0.019s
I20261116 08:58:03.223417 20703 raft_consensus.cc:2826] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:03.224416 20699 tablet_bootstrap.cc:654] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:03.224269 20697 raft_consensus.cc:360] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.225006 20697 raft_consensus.cc:386] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:03.225366 20697 raft_consensus.cc:748] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Initialized, Role: FOLLOWER
I20261116 08:58:03.226225 20697 consensus_queue.cc:261] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.227077 20698 raft_consensus.cc:360] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.228111 20698 raft_consensus.cc:386] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:03.228529 20698 raft_consensus.cc:748] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92186d919b741b4b8d69416caad3021, State: Initialized, Role: FOLLOWER
I20261116 08:58:03.229645 20698 consensus_queue.cc:261] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.232571 20703 raft_consensus.cc:705] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 1 LEADER]: Becoming Leader. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Running, Role: LEADER
I20261116 08:58:03.233681 20703 consensus_queue.cc:238] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.236325 20697 ts_tablet_manager.cc:1480] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20261116 08:58:03.237296 20699 tablet_bootstrap.cc:492] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa: No bootstrap required, opened a new log
I20261116 08:58:03.240214 20699 ts_tablet_manager.cc:1449] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.013s
I20261116 08:58:03.242882 20698 ts_tablet_manager.cc:1480] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20261116 08:58:03.246553 20699 raft_consensus.cc:360] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.247298 20699 raft_consensus.cc:386] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:03.247609 20699 raft_consensus.cc:748] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5439f672b0b4560aa83c325f85cc6aa, State: Initialized, Role: FOLLOWER
I20261116 08:58:03.247681 20709 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:03.248230 20709 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.248435 20699 consensus_queue.cc:261] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.250630 20709 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:03.251207 20697 tablet_bootstrap.cc:492] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850: Bootstrap starting.
I20261116 08:58:03.251444 20495 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:03.252099 20495 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 0.
I20261116 08:58:03.253480 20536 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850; no voters:
I20261116 08:58:03.254320 20709 raft_consensus.cc:2826] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 08:58:03.254693 20709 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 08:58:03.255030 20709 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:03.255926 20651 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:03.256567 20651 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 0.
I20261116 08:58:03.255460 20699 ts_tablet_manager.cc:1480] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20261116 08:58:03.260108 20699 tablet_bootstrap.cc:492] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa: Bootstrap starting.
I20261116 08:58:03.264384 20709 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.265297 20698 tablet_bootstrap.cc:492] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021: Bootstrap starting.
I20261116 08:58:03.266820 20699 tablet_bootstrap.cc:654] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:03.269656 20495 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021"
I20261116 08:58:03.270339 20697 tablet_bootstrap.cc:654] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:03.270304 20495 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:03.273238 20698 tablet_bootstrap.cc:654] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:03.291489 20226 catalog_manager.cc:5840] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 reported cstate change: term changed from 0 to 1, leader changed from <none> to e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2). New cstate: current_term: 1 leader_uuid: "e950e37c4d2d425a9771e063fdaa2850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20261116 08:58:03.295794 20412 catalog_manager.cc:1310] Loaded cluster ID: f1b7bc8047cd4836a24134372a439417
W20261116 08:58:03.298501 20538 consensus_peers.cc:597] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 -> Peer f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323): Couldn't send request to peer f5439f672b0b4560aa83c325f85cc6aa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:03.300863 20412 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: loading cluster ID for follower catalog manager: success
I20261116 08:58:03.303049 20651 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
I20261116 08:58:03.305300 20651 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:03.304296 20709 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 1 election: Requested vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:03.307744 20711 consensus_queue.cc:1049] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261116 08:58:03.313062 20495 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 1.
I20261116 08:58:03.318776 20536 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850; no voters:
I20261116 08:58:03.321280 20711 raft_consensus.cc:2826] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:03.325740 20711 raft_consensus.cc:705] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 1 LEADER]: Becoming Leader. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Running, Role: LEADER
I20261116 08:58:03.329424 20651 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 1.
I20261116 08:58:03.330756 20698 tablet_bootstrap.cc:492] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021: No bootstrap required, opened a new log
I20261116 08:58:03.331287 20698 ts_tablet_manager.cc:1449] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.021s
I20261116 08:58:03.332448 20412 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: acquiring CA information for follower catalog manager: success
I20261116 08:58:03.327000 20711 consensus_queue.cc:238] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.337291 20697 tablet_bootstrap.cc:492] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850: No bootstrap required, opened a new log
I20261116 08:58:03.337760 20697 ts_tablet_manager.cc:1449] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.011s
I20261116 08:58:03.344633 20412 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261116 08:58:03.354485 20438 catalog_manager.cc:1310] Loaded cluster ID: f1b7bc8047cd4836a24134372a439417
I20261116 08:58:03.354733 20438 catalog_manager.cc:1603] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: loading cluster ID for follower catalog manager: success
I20261116 08:58:03.356549 20698 raft_consensus.cc:360] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.357256 20698 raft_consensus.cc:386] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:03.357542 20698 raft_consensus.cc:748] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92186d919b741b4b8d69416caad3021, State: Initialized, Role: FOLLOWER
I20261116 08:58:03.358222 20698 consensus_queue.cc:261] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.360513 20698 ts_tablet_manager.cc:1480] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20261116 08:58:03.363610 20438 catalog_manager.cc:1625] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: acquiring CA information for follower catalog manager: success
I20261116 08:58:03.366600 20699 tablet_bootstrap.cc:492] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa: No bootstrap required, opened a new log
I20261116 08:58:03.367060 20699 ts_tablet_manager.cc:1449] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa: Time spent bootstrapping tablet: real 0.107s user 0.022s sys 0.017s
I20261116 08:58:03.363025 20697 raft_consensus.cc:360] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.368592 20697 raft_consensus.cc:386] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:03.368961 20697 raft_consensus.cc:748] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Initialized, Role: FOLLOWER
I20261116 08:58:03.369884 20699 raft_consensus.cc:360] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.370627 20699 raft_consensus.cc:386] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:03.370019 20697 consensus_queue.cc:261] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.372822 20697 ts_tablet_manager.cc:1480] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850: Time spent starting tablet: real 0.034s user 0.005s sys 0.003s
I20261116 08:58:03.374505 20704 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:03.370929 20699 raft_consensus.cc:748] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5439f672b0b4560aa83c325f85cc6aa, State: Initialized, Role: FOLLOWER
I20261116 08:58:03.380541 20704 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.383252 20704 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:03.385397 20699 consensus_queue.cc:261] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.392779 20438 catalog_manager.cc:1653] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261116 08:58:03.395625 20699 ts_tablet_manager.cc:1480] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20261116 08:58:03.402125 20225 catalog_manager.cc:5840] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 reported cstate change: term changed from 0 to 1, leader changed from <none> to e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2). New cstate: current_term: 1 leader_uuid: "e950e37c4d2d425a9771e063fdaa2850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20261116 08:58:03.407871 20703 consensus_queue.cc:1049] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261116 08:58:03.427906 20706 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:03.428385 20706 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.430298 20706 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:03.431949 20573 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:03.432583 20573 raft_consensus.cc:2490] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5439f672b0b4560aa83c325f85cc6aa in term 0.
I20261116 08:58:03.433914 20616 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa; no voters:
I20261116 08:58:03.434796 20706 raft_consensus.cc:2826] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 08:58:03.450307 20706 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 08:58:03.450671 20706 raft_consensus.cc:3096] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:03.456091 20495 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:03.456003 20716 consensus_queue.cc:1049] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20261116 08:58:03.466768 20706 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.456776 20495 raft_consensus.cc:2490] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5439f672b0b4560aa83c325f85cc6aa in term 0.
I20261116 08:58:03.453881 20705 raft_consensus.cc:499] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:03.486122 20495 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021"
I20261116 08:58:03.486895 20495 raft_consensus.cc:3096] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:03.493458 20705 raft_consensus.cc:521] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.495621 20650 raft_consensus.cc:3096] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:03.505146 20573 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850"
I20261116 08:58:03.507565 20573 raft_consensus.cc:3096] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:03.507963 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:03.504092 20706 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 1 election: Requested vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:03.514190 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:03.514681 20705 leader_election.cc:290] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:03.515825 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:03.521623 20614 leader_election.cc:304] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5439f672b0b4560aa83c325f85cc6aa; no voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850
I20261116 08:58:03.526944 20573 raft_consensus.cc:2490] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5439f672b0b4560aa83c325f85cc6aa in term 1.
I20261116 08:58:03.529120 20616 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa; no voters:
I20261116 08:58:03.556820 20495 raft_consensus.cc:2490] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5439f672b0b4560aa83c325f85cc6aa in term 1.
I20261116 08:58:03.533531 20706 raft_consensus.cc:2826] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:03.561117 20706 raft_consensus.cc:705] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 1 LEADER]: Becoming Leader. State: Replica: f5439f672b0b4560aa83c325f85cc6aa, State: Running, Role: LEADER
I20261116 08:58:03.562397 20706 consensus_queue.cc:238] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.577291 20649 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:03.579311 20459 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d92186d919b741b4b8d69416caad3021; no voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:03.580186 20704 raft_consensus.cc:2771] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261116 08:58:03.586336 20706 consensus_queue.cc:1049] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 08:58:03.598107 20726 consensus_queue.cc:1049] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261116 08:58:03.616415 20225 catalog_manager.cc:5840] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3). New cstate: current_term: 1 leader_uuid: "f5439f672b0b4560aa83c325f85cc6aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } } }
I20261116 08:58:03.624182 20705 raft_consensus.cc:2771] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261116 08:58:03.880906 20573 tablet_service.cc:2061] Received LeaderStepDown RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1"
dest_uuid: "e950e37c4d2d425a9771e063fdaa2850"
from {username='slave'} at 127.0.0.1:38444
I20261116 08:58:03.881528 20573 raft_consensus.cc:597] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 1 LEADER]: Received request to step down
I20261116 08:58:03.881810 20573 raft_consensus.cc:3091] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 1 LEADER]: Stepping down as leader of term 1
I20261116 08:58:03.882047 20573 raft_consensus.cc:748] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Running, Role: LEADER
I20261116 08:58:03.882647 20573 consensus_queue.cc:261] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.883702 20573 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:03.985637 20705 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e950e37c4d2d425a9771e063fdaa2850)
I20261116 08:58:03.986109 20705 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.987823 20705 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:03.988618 20495 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:03.988915 20573 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:03.989513 20573 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5439f672b0b4560aa83c325f85cc6aa in term 2.
I20261116 08:58:03.990408 20616 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa; no voters: d92186d919b741b4b8d69416caad3021
I20261116 08:58:03.990953 20705 raft_consensus.cc:2826] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261116 08:58:03.991276 20705 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Starting leader election (detected failure of leader e950e37c4d2d425a9771e063fdaa2850)
I20261116 08:58:03.991556 20705 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:03.996255 20705 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:03.998394 20705 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 2 election: Requested vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:03.999218 20495 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021"
I20261116 08:58:03.999451 20573 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850"
I20261116 08:58:04.004184 20573 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5439f672b0b4560aa83c325f85cc6aa in term 2.
I20261116 08:58:04.005046 20616 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa; no voters: d92186d919b741b4b8d69416caad3021
I20261116 08:58:04.005681 20705 raft_consensus.cc:2826] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Leader election won for term 2
I20261116 08:58:04.006067 20705 raft_consensus.cc:705] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 2 LEADER]: Becoming Leader. State: Replica: f5439f672b0b4560aa83c325f85cc6aa, State: Running, Role: LEADER
I20261116 08:58:04.006748 20705 consensus_queue.cc:238] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:04.010880 20495 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:04.011108 20573 raft_consensus.cc:1285] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Refusing update from remote peer f5439f672b0b4560aa83c325f85cc6aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261116 08:58:04.015100 20224 catalog_manager.cc:5840] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa reported cstate change: term changed from 1 to 2, leader changed from e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2) to f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3). New cstate: current_term: 2 leader_uuid: "f5439f672b0b4560aa83c325f85cc6aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } health_report { overall_health: HEALTHY } } }
I20261116 08:58:04.016424 20705 consensus_queue.cc:1049] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261116 08:58:04.017637 20495 raft_consensus.cc:1285] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Refusing update from remote peer f5439f672b0b4560aa83c325f85cc6aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261116 08:58:04.018920 20705 consensus_queue.cc:1049] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261116 08:58:04.087198 20573 tablet_service.cc:2061] Received LeaderStepDown RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9"
dest_uuid: "e950e37c4d2d425a9771e063fdaa2850"
from {username='slave'} at 127.0.0.1:38444
I20261116 08:58:04.087854 20573 raft_consensus.cc:597] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 1 LEADER]: Received request to step down
I20261116 08:58:04.088214 20573 raft_consensus.cc:3091] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 1 LEADER]: Stepping down as leader of term 1
I20261116 08:58:04.088517 20573 raft_consensus.cc:748] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Running, Role: LEADER
I20261116 08:58:04.089632 20573 consensus_queue.cc:261] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:04.090824 20573 raft_consensus.cc:3096] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:04.268761 20705 raft_consensus.cc:499] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e950e37c4d2d425a9771e063fdaa2850)
I20261116 08:58:04.269631 20705 raft_consensus.cc:521] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:04.273887 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:04.274417 20573 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:04.275197 20573 raft_consensus.cc:2490] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5439f672b0b4560aa83c325f85cc6aa in term 2.
I20261116 08:58:04.275403 20705 leader_election.cc:290] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:04.277084 20616 leader_election.cc:304] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa; no voters:
I20261116 08:58:04.278080 20705 raft_consensus.cc:2826] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261116 08:58:04.278443 20705 raft_consensus.cc:499] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Starting leader election (detected failure of leader e950e37c4d2d425a9771e063fdaa2850)
I20261116 08:58:04.278779 20705 raft_consensus.cc:3096] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:04.281025 20704 raft_consensus.cc:499] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e950e37c4d2d425a9771e063fdaa2850)
I20261116 08:58:04.281549 20704 raft_consensus.cc:521] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:04.283566 20704 leader_election.cc:290] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:04.285462 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:04.285722 20705 raft_consensus.cc:521] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:04.286898 20650 raft_consensus.cc:2415] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d92186d919b741b4b8d69416caad3021 in current term 2: Already voted for candidate f5439f672b0b4560aa83c325f85cc6aa in this term.
I20261116 08:58:04.286842 20573 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:04.287649 20705 leader_election.cc:290] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 2 election: Requested vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:04.288178 20573 raft_consensus.cc:2490] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92186d919b741b4b8d69416caad3021 in term 2.
I20261116 08:58:04.292443 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021"
I20261116 08:58:04.293017 20494 raft_consensus.cc:3096] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:04.298381 20459 leader_election.cc:304] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850; no voters: f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:04.311539 20494 raft_consensus.cc:2490] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5439f672b0b4560aa83c325f85cc6aa in term 2.
I20261116 08:58:04.313298 20704 raft_consensus.cc:2786] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20261116 08:58:04.315332 20614 leader_election.cc:304] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d92186d919b741b4b8d69416caad3021, f5439f672b0b4560aa83c325f85cc6aa; no voters:
I20261116 08:58:04.317092 20706 raft_consensus.cc:2826] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Leader election won for term 2
I20261116 08:58:04.317620 20706 raft_consensus.cc:705] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 2 LEADER]: Becoming Leader. State: Replica: f5439f672b0b4560aa83c325f85cc6aa, State: Running, Role: LEADER
I20261116 08:58:04.318660 20706 consensus_queue.cc:238] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
W20261116 08:58:04.483323 20614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.102.1:37237, user_credentials={real_user=slave}} blocked reactor thread for 168040us
I20261116 08:58:04.563973 20748 raft_consensus.cc:499] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9a31f175a1a4361aace9c508bdb5291)
I20261116 08:58:04.564636 20748 raft_consensus.cc:521] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:04.567695 20237 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9188aed91ff24b0ea766bf89f306b495" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d9a31f175a1a4361aace9c508bdb5291" is_pre_election: true
I20261116 08:58:04.572399 20748 leader_election.cc:290] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9a31f175a1a4361aace9c508bdb5291 (127.18.102.62:32983), a356758e034e49e292899b82f51810cd (127.18.102.60:32843)
I20261116 08:58:04.609556 20366 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9188aed91ff24b0ea766bf89f306b495" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "a356758e034e49e292899b82f51810cd" is_pre_election: true
I20261116 08:58:04.610347 20366 raft_consensus.cc:2490] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9188aed91ff24b0ea766bf89f306b495 in term 1.
I20261116 08:58:04.612510 20276 leader_election.cc:304] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9188aed91ff24b0ea766bf89f306b495, a356758e034e49e292899b82f51810cd; no voters: d9a31f175a1a4361aace9c508bdb5291
I20261116 08:58:04.613641 20748 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261116 08:58:04.614053 20748 raft_consensus.cc:499] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d9a31f175a1a4361aace9c508bdb5291)
I20261116 08:58:04.614434 20748 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:04.621948 20748 raft_consensus.cc:521] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:04.623997 20748 leader_election.cc:290] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [CANDIDATE]: Term 2 election: Requested vote from peers d9a31f175a1a4361aace9c508bdb5291 (127.18.102.62:32983), a356758e034e49e292899b82f51810cd (127.18.102.60:32843)
I20261116 08:58:04.625341 20237 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9188aed91ff24b0ea766bf89f306b495" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d9a31f175a1a4361aace9c508bdb5291"
I20261116 08:58:04.626202 20366 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9188aed91ff24b0ea766bf89f306b495" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "a356758e034e49e292899b82f51810cd"
I20261116 08:58:04.627123 20366 raft_consensus.cc:3096] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:04.633713 20366 raft_consensus.cc:2490] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9188aed91ff24b0ea766bf89f306b495 in term 2.
I20261116 08:58:04.635133 20276 leader_election.cc:304] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9188aed91ff24b0ea766bf89f306b495, a356758e034e49e292899b82f51810cd; no voters: d9a31f175a1a4361aace9c508bdb5291
I20261116 08:58:04.638975 20748 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 2 FOLLOWER]: Leader election won for term 2
I20261116 08:58:04.640105 20748 raft_consensus.cc:705] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 2 LEADER]: Becoming Leader. State: Replica: 9188aed91ff24b0ea766bf89f306b495, State: Running, Role: LEADER
I20261116 08:58:04.640981 20748 consensus_queue.cc:238] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:04.644569 20302 raft_consensus.cc:1250] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 2 LEADER]: Rejecting Update request from peer d9a31f175a1a4361aace9c508bdb5291 for earlier term 1. Current term is 2. Ops: []
I20261116 08:58:04.648571 20366 raft_consensus.cc:1285] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 2 FOLLOWER]: Refusing update from remote peer 9188aed91ff24b0ea766bf89f306b495: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20261116 08:58:04.648749 20237 raft_consensus.cc:3091] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 1 LEADER]: Stepping down as leader of term 1
I20261116 08:58:04.649129 20237 raft_consensus.cc:748] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9a31f175a1a4361aace9c508bdb5291, State: Running, Role: LEADER
I20261116 08:58:04.649905 20237 consensus_queue.cc:261] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:04.650262 20751 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20261116 08:58:04.650883 20237 raft_consensus.cc:3096] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:04.655828 20753 sys_catalog.cc:455] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9188aed91ff24b0ea766bf89f306b495. Latest consensus state: current_term: 2 leader_uuid: "9188aed91ff24b0ea766bf89f306b495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:04.656562 20753 sys_catalog.cc:458] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:04.660765 20237 raft_consensus.cc:1285] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 2 FOLLOWER]: Refusing update from remote peer 9188aed91ff24b0ea766bf89f306b495: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20261116 08:58:04.663195 20751 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20261116 08:58:04.678028 20748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9188aed91ff24b0ea766bf89f306b495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:04.678759 20748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:04.680104 20755 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:04.680249 20756 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:58:04.680622 20755 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:04.683466 20755 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
W20261116 08:58:04.710126 20211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20261116 08:58:04.724509 20214 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20261116 08:58:04.727864 20750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9188aed91ff24b0ea766bf89f306b495. Latest consensus state: current_term: 2 leader_uuid: "9188aed91ff24b0ea766bf89f306b495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:04.728700 20750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:04.733559 20757 raft_consensus.cc:499] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:04.734112 20758 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:04.734107 20757 raft_consensus.cc:521] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:04.734747 20758 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:04.736738 20758 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:04.737211 20758 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:04.737291 20757 leader_election.cc:290] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:04.734548 20747 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9188aed91ff24b0ea766bf89f306b495. Latest consensus state: current_term: 2 leader_uuid: "9188aed91ff24b0ea766bf89f306b495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:04.737682 20758 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:04.738198 20747 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:04.738492 20573 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850"
I20261116 08:58:04.738807 20747 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9188aed91ff24b0ea766bf89f306b495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:04.739518 20747 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:04.739665 20758 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:04.740439 20651 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:04.746287 20651 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:04.747639 20571 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:04.748342 20651 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:04.749607 20573 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850"
I20261116 08:58:04.749657 20459 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d92186d919b741b4b8d69416caad3021; no voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:04.750555 20755 raft_consensus.cc:2771] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20261116 08:58:04.757627 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:04.758311 20494 raft_consensus.cc:1285] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Refusing update from remote peer f5439f672b0b4560aa83c325f85cc6aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261116 08:58:04.761951 20749 consensus_queue.cc:1049] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20261116 08:58:04.770093 20495 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:04.770370 20493 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:04.771580 20492 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:04.774281 20536 leader_election.cc:304] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850; no voters: d92186d919b741b4b8d69416caad3021, f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:04.775529 20536 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850; no voters: d92186d919b741b4b8d69416caad3021, f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:04.776629 20536 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850; no voters: d92186d919b741b4b8d69416caad3021, f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:04.778115 20703 raft_consensus.cc:2771] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20261116 08:58:04.778733 20703 raft_consensus.cc:2771] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20261116 08:58:04.779306 20703 raft_consensus.cc:2771] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261116 08:58:04.787835 20756 catalog_manager.cc:694] Loaded metadata for table test-table [id=57a542ce297949569849767182f5b9d0]
W20261116 08:58:04.791221 20211 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20261116 08:58:04.801563 20650 tablet_service.cc:2061] Received LeaderStepDown RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9"
dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
from {username='slave'} at 127.0.0.1:42472
I20261116 08:58:04.802161 20650 raft_consensus.cc:597] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 2 LEADER]: Received request to step down
I20261116 08:58:04.802482 20650 raft_consensus.cc:3091] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 2 LEADER]: Stepping down as leader of term 2
I20261116 08:58:04.802758 20650 raft_consensus.cc:748] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f5439f672b0b4560aa83c325f85cc6aa, State: Running, Role: LEADER
I20261116 08:58:04.803491 20650 consensus_queue.cc:261] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:04.804164 20650 raft_consensus.cc:3096] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:58:04.809216 20367 raft_consensus.cc:1250] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 2 FOLLOWER]: Rejecting Update request from peer d9a31f175a1a4361aace9c508bdb5291 for earlier term 1. Current term is 2. Ops: []
I20261116 08:58:04.811182 20750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9188aed91ff24b0ea766bf89f306b495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:04.811888 20750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: This master's current role is: LEADER
W20261116 08:58:04.817708 20214 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20261116 08:58:04.824352 20753 sys_catalog.cc:455] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9188aed91ff24b0ea766bf89f306b495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:04.824899 20753 sys_catalog.cc:458] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:04.829022 20756 tablet_loader.cc:96] loaded metadata for tablet 19ac1f3e0f6c4b37b015139ac43a746a (table test-table [id=57a542ce297949569849767182f5b9d0])
I20261116 08:58:04.830960 20756 tablet_loader.cc:96] loaded metadata for tablet 8663ba9cebca4dd0922589c3095d07f1 (table test-table [id=57a542ce297949569849767182f5b9d0])
I20261116 08:58:04.838464 20756 tablet_loader.cc:96] loaded metadata for tablet 86badd1d1e29431daebce9a9ae80f0c9 (table test-table [id=57a542ce297949569849767182f5b9d0])
I20261116 08:58:04.840075 20756 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:58:04.844915 20756 catalog_manager.cc:1310] Loaded cluster ID: f1b7bc8047cd4836a24134372a439417
I20261116 08:58:04.845216 20756 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:58:04.850311 20756 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:58:04.855185 20756 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: Loaded TSK: 0
I20261116 08:58:04.856364 20756 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:58:04.990674 20439 catalog_manager.cc:1653] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261116 08:58:04.993953 20757 raft_consensus.cc:499] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:04.994474 20757 raft_consensus.cc:521] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:04.996443 20757 leader_election.cc:290] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:04.996938 20704 raft_consensus.cc:499] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:04.997331 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:04.997520 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:04.997422 20704 raft_consensus.cc:521] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:04.998191 20650 raft_consensus.cc:2490] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 3.
I20261116 08:58:04.998471 20494 raft_consensus.cc:2490] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 2.
I20261116 08:58:04.999466 20704 leader_election.cc:290] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:05.001315 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:04.999776 20538 leader_election.cc:304] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa; no voters:
I20261116 08:58:05.001927 20572 raft_consensus.cc:2490] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92186d919b741b4b8d69416caad3021 in term 2.
I20261116 08:58:05.003031 20459 leader_election.cc:304] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850; no voters:
I20261116 08:58:05.004117 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:05.004686 20650 raft_consensus.cc:2490] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92186d919b741b4b8d69416caad3021 in term 3.
I20261116 08:58:05.005733 20757 raft_consensus.cc:2826] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261116 08:58:05.006094 20757 raft_consensus.cc:499] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:05.006366 20757 raft_consensus.cc:3096] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:58:05.012702 20757 raft_consensus.cc:521] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.013399 20677 heartbeater.cc:500] Master 127.18.102.61:36365 was elected leader, sending a full tablet report...
I20261116 08:58:05.014441 20599 heartbeater.cc:500] Master 127.18.102.61:36365 was elected leader, sending a full tablet report...
I20261116 08:58:05.015923 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
I20261116 08:58:05.016239 20521 heartbeater.cc:500] Master 127.18.102.61:36365 was elected leader, sending a full tablet report...
I20261116 08:58:05.005378 20704 raft_consensus.cc:2826] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261116 08:58:05.018556 20704 raft_consensus.cc:499] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:05.018968 20704 raft_consensus.cc:3096] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:58:05.021729 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021"
I20261116 08:58:05.032958 20650 raft_consensus.cc:2490] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 3.
I20261116 08:58:05.038190 20538 leader_election.cc:304] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa; no voters:
I20261116 08:58:05.038796 20757 leader_election.cc:290] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 election: Requested vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:05.038975 20758 raft_consensus.cc:2826] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 3 FOLLOWER]: Leader election won for term 3
I20261116 08:58:05.037127 20704 raft_consensus.cc:521] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.039515 20758 raft_consensus.cc:705] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 3 LEADER]: Becoming Leader. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Running, Role: LEADER
I20261116 08:58:05.040365 20758 consensus_queue.cc:238] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.041502 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850"
I20261116 08:58:05.041985 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
I20261116 08:58:05.042693 20650 raft_consensus.cc:2415] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d92186d919b741b4b8d69416caad3021 in current term 3: Already voted for candidate e950e37c4d2d425a9771e063fdaa2850 in this term.
I20261116 08:58:05.043627 20704 leader_election.cc:290] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 3 election: Requested vote from peers e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:05.044229 20459 leader_election.cc:304] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d92186d919b741b4b8d69416caad3021; no voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:05.046448 20650 raft_consensus.cc:1285] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Refusing update from remote peer e950e37c4d2d425a9771e063fdaa2850: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261116 08:58:05.048367 20703 consensus_queue.cc:1049] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20261116 08:58:05.052870 20494 raft_consensus.cc:2415] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e950e37c4d2d425a9771e063fdaa2850 in current term 3: Already voted for candidate d92186d919b741b4b8d69416caad3021 in this term.
I20261116 08:58:05.053290 20704 raft_consensus.cc:2771] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20261116 08:58:05.057850 20492 raft_consensus.cc:1285] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Refusing update from remote peer e950e37c4d2d425a9771e063fdaa2850: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261116 08:58:05.069865 20758 consensus_queue.cc:1049] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261116 08:58:05.072759 20292 catalog_manager.cc:5840] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "e950e37c4d2d425a9771e063fdaa2850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20261116 08:58:05.084771 20650 tablet_service.cc:2061] Received LeaderStepDown RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1"
dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
from {username='slave'} at 127.0.0.1:42472
I20261116 08:58:05.085423 20650 raft_consensus.cc:597] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 2 LEADER]: Received request to step down
I20261116 08:58:05.085736 20650 raft_consensus.cc:3091] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 2 LEADER]: Stepping down as leader of term 2
I20261116 08:58:05.091472 20650 raft_consensus.cc:748] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f5439f672b0b4560aa83c325f85cc6aa, State: Running, Role: LEADER
I20261116 08:58:05.092423 20650 consensus_queue.cc:261] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.094126 20650 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:58:05.236730 20758 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:05.237262 20758 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.238993 20758 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:05.239892 20704 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:05.239964 20492 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:05.240141 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:05.240485 20704 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.240940 20650 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 3.
I20261116 08:58:05.241472 20492 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 2.
I20261116 08:58:05.242004 20538 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa; no voters:
I20261116 08:58:05.242533 20704 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:05.242838 20758 raft_consensus.cc:2826] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261116 08:58:05.243220 20758 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:05.243590 20758 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:58:05.243841 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:05.243989 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:05.244459 20650 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92186d919b741b4b8d69416caad3021 in term 3.
I20261116 08:58:05.245497 20459 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d92186d919b741b4b8d69416caad3021, f5439f672b0b4560aa83c325f85cc6aa; no voters:
I20261116 08:58:05.246122 20704 raft_consensus.cc:2826] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261116 08:58:05.246490 20704 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:05.246893 20704 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:58:05.250432 20758 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.251516 20572 raft_consensus.cc:2415] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d92186d919b741b4b8d69416caad3021 in current term 3: Already voted for candidate e950e37c4d2d425a9771e063fdaa2850 in this term.
I20261116 08:58:05.252866 20758 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 election: Requested vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:05.253762 20492 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021"
I20261116 08:58:05.253635 20704 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.254007 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
I20261116 08:58:05.254807 20492 raft_consensus.cc:2415] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e950e37c4d2d425a9771e063fdaa2850 in current term 3: Already voted for candidate d92186d919b741b4b8d69416caad3021 in this term.
I20261116 08:58:05.255709 20704 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 3 election: Requested vote from peers e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:05.256889 20651 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
I20261116 08:58:05.257094 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850"
I20261116 08:58:05.257756 20572 raft_consensus.cc:2415] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d92186d919b741b4b8d69416caad3021 in current term 3: Already voted for candidate e950e37c4d2d425a9771e063fdaa2850 in this term.
I20261116 08:58:05.258785 20459 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d92186d919b741b4b8d69416caad3021; no voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:05.259559 20704 raft_consensus.cc:2771] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20261116 08:58:05.261053 20650 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 3.
I20261116 08:58:05.262012 20538 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa; no voters: d92186d919b741b4b8d69416caad3021
I20261116 08:58:05.262718 20758 raft_consensus.cc:2826] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 3 FOLLOWER]: Leader election won for term 3
I20261116 08:58:05.263253 20758 raft_consensus.cc:705] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 3 LEADER]: Becoming Leader. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Running, Role: LEADER
I20261116 08:58:05.264071 20758 consensus_queue.cc:238] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.269778 20492 raft_consensus.cc:1285] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Refusing update from remote peer e950e37c4d2d425a9771e063fdaa2850: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261116 08:58:05.271855 20650 raft_consensus.cc:1285] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Refusing update from remote peer e950e37c4d2d425a9771e063fdaa2850: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261116 08:58:05.271952 20290 catalog_manager.cc:5840] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 reported cstate change: term changed from 2 to 3, leader changed from f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3) to e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2). New cstate: current_term: 3 leader_uuid: "e950e37c4d2d425a9771e063fdaa2850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20261116 08:58:05.273507 20758 consensus_queue.cc:1049] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261116 08:58:05.274510 20766 consensus_queue.cc:1049] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20261116 08:58:05.299013 20651 tablet_service.cc:2061] Received LeaderStepDown RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a"
dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
from {username='slave'} at 127.0.0.1:42472
I20261116 08:58:05.299705 20651 raft_consensus.cc:597] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 1 LEADER]: Received request to step down
I20261116 08:58:05.300071 20651 raft_consensus.cc:3091] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 1 LEADER]: Stepping down as leader of term 1
I20261116 08:58:05.300367 20651 raft_consensus.cc:748] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f5439f672b0b4560aa83c325f85cc6aa, State: Running, Role: LEADER
I20261116 08:58:05.301165 20651 consensus_queue.cc:261] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.302369 20651 raft_consensus.cc:3096] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:05.467677 20704 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:05.468076 20704 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.469532 20704 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:05.470299 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:05.470587 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:05.471220 20650 raft_consensus.cc:2490] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92186d919b741b4b8d69416caad3021 in term 2.
I20261116 08:58:05.472270 20459 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d92186d919b741b4b8d69416caad3021, f5439f672b0b4560aa83c325f85cc6aa; no voters: e950e37c4d2d425a9771e063fdaa2850
I20261116 08:58:05.472883 20704 raft_consensus.cc:2826] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261116 08:58:05.473184 20704 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:05.473448 20704 raft_consensus.cc:3096] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:05.477810 20704 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.479354 20704 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 2 election: Requested vote from peers e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:05.480111 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850"
I20261116 08:58:05.480414 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
I20261116 08:58:05.484820 20650 raft_consensus.cc:2490] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92186d919b741b4b8d69416caad3021 in term 2.
I20261116 08:58:05.485637 20459 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d92186d919b741b4b8d69416caad3021, f5439f672b0b4560aa83c325f85cc6aa; no voters: e950e37c4d2d425a9771e063fdaa2850
I20261116 08:58:05.486246 20704 raft_consensus.cc:2826] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Leader election won for term 2
I20261116 08:58:05.486578 20704 raft_consensus.cc:705] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 2 LEADER]: Becoming Leader. State: Replica: d92186d919b741b4b8d69416caad3021, State: Running, Role: LEADER
I20261116 08:58:05.487319 20704 consensus_queue.cc:238] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.493944 20757 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f5439f672b0b4560aa83c325f85cc6aa)
I20261116 08:58:05.494434 20757 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:05.496186 20572 raft_consensus.cc:3096] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:05.496445 20757 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:05.497095 20650 raft_consensus.cc:1285] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Refusing update from remote peer d92186d919b741b4b8d69416caad3021: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261116 08:58:05.497987 20651 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:05.498716 20651 raft_consensus.cc:2415] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e950e37c4d2d425a9771e063fdaa2850 in current term 2: Already voted for candidate d92186d919b741b4b8d69416caad3021 in this term.
I20261116 08:58:05.499689 20492 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:05.501345 20536 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850; no voters: d92186d919b741b4b8d69416caad3021, f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:05.501791 20755 consensus_queue.cc:1049] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261116 08:58:05.503460 20292 catalog_manager.cc:5840] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 reported cstate change: term changed from 1 to 2, leader changed from f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3) to d92186d919b741b4b8d69416caad3021 (127.18.102.1). New cstate: current_term: 2 leader_uuid: "d92186d919b741b4b8d69416caad3021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20261116 08:58:05.504545 20572 raft_consensus.cc:1285] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Refusing update from remote peer d92186d919b741b4b8d69416caad3021: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261116 08:58:05.505067 20757 raft_consensus.cc:2771] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261116 08:58:05.505971 20755 consensus_queue.cc:1049] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261116 08:58:05.561851 20290 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40970:
table { table_id: "57a542ce297949569849767182f5b9d0" table_name: "test-table" } new_table_name: "NewTableName"
W20261116 08:58:05.578198 20258 debug-util.cc:398] Leaking SignalData structure 0x7b08000883c0 after lost signal to thread 18843
W20261116 08:58:05.579457 20258 debug-util.cc:398] Leaking SignalData structure 0x7b08000ff260 after lost signal to thread 20261
W20261116 08:58:05.581056 20258 debug-util.cc:398] Leaking SignalData structure 0x7b08001d68a0 after lost signal to thread 20391
I20261116 08:58:05.584374 18840 master.cc:585] Master@127.18.102.61:36365 shutting down...
I20261116 08:58:05.694444 20788 tablet.cc:1759] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261116 08:58:05.696913 20789 tablet.cc:1759] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261116 08:58:05.700587 20787 tablet.cc:1759] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261116 08:58:05.711975 20791 tablet.cc:1759] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261116 08:58:05.723224 20792 tablet.cc:1759] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261116 08:58:05.731282 20793 tablet.cc:1759] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261116 08:58:05.726676 20787 tablet.cc:1759] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261116 08:58:05.747486 20790 tablet.cc:1759] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261116 08:58:05.735054 20794 tablet.cc:1759] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261116 08:58:05.795240 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 2 LEADER]: Raft consensus shutting down.
I20261116 08:58:05.796213 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:05.796672 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: stopping tablet replica
W20261116 08:58:05.809443 18840 connection.cc:466] server connection from 127.0.0.1:40944 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40944 (request call id 40) could send its response
I20261116 08:58:05.825244 18840 master.cc:607] Master@127.18.102.61:36365 shutdown complete.
I20261116 08:58:05.896293 20797 raft_consensus.cc:499] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9188aed91ff24b0ea766bf89f306b495)
I20261116 08:58:05.896711 20797 raft_consensus.cc:521] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:05.898209 20797 leader_election.cc:290] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9188aed91ff24b0ea766bf89f306b495 (127.18.102.61:36365), a356758e034e49e292899b82f51810cd (127.18.102.60:32843)
I20261116 08:58:05.899334 20367 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9a31f175a1a4361aace9c508bdb5291" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "a356758e034e49e292899b82f51810cd" is_pre_election: true
W20261116 08:58:05.905570 20214 leader_election.cc:336] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9188aed91ff24b0ea766bf89f306b495 (127.18.102.61:36365): Network error: Client connection negotiation failed: client connection to 127.18.102.61:36365: connect: Connection refused (error 111)
I20261116 08:58:05.906137 20214 leader_election.cc:304] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9a31f175a1a4361aace9c508bdb5291; no voters: 9188aed91ff24b0ea766bf89f306b495, a356758e034e49e292899b82f51810cd
I20261116 08:58:05.911087 20797 raft_consensus.cc:2771] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20261116 08:58:05.982264 20802 raft_consensus.cc:499] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9188aed91ff24b0ea766bf89f306b495)
I20261116 08:58:05.982714 20802 raft_consensus.cc:521] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:05.985567 20237 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a356758e034e49e292899b82f51810cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "d9a31f175a1a4361aace9c508bdb5291" is_pre_election: true
I20261116 08:58:05.986337 20237 raft_consensus.cc:2490] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a356758e034e49e292899b82f51810cd in term 2.
I20261116 08:58:05.986466 20802 leader_election.cc:290] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9a31f175a1a4361aace9c508bdb5291 (127.18.102.62:32983), 9188aed91ff24b0ea766bf89f306b495 (127.18.102.61:36365)
I20261116 08:58:05.987746 20341 leader_election.cc:304] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a356758e034e49e292899b82f51810cd, d9a31f175a1a4361aace9c508bdb5291; no voters:
I20261116 08:58:05.989197 20802 raft_consensus.cc:2826] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261116 08:58:05.991066 20802 raft_consensus.cc:499] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9188aed91ff24b0ea766bf89f306b495)
I20261116 08:58:05.991513 20802 raft_consensus.cc:3096] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 2 FOLLOWER]: Advancing to term 3
W20261116 08:58:05.996222 20344 leader_election.cc:336] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9188aed91ff24b0ea766bf89f306b495 (127.18.102.61:36365): Network error: Client connection negotiation failed: client connection to 127.18.102.61:36365: connect: Connection refused (error 111)
I20261116 08:58:05.998842 20802 raft_consensus.cc:521] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:06.001132 20802 leader_election.cc:290] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [CANDIDATE]: Term 3 election: Requested vote from peers d9a31f175a1a4361aace9c508bdb5291 (127.18.102.62:32983), 9188aed91ff24b0ea766bf89f306b495 (127.18.102.61:36365)
I20261116 08:58:06.001989 20237 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a356758e034e49e292899b82f51810cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "d9a31f175a1a4361aace9c508bdb5291"
I20261116 08:58:06.002595 20237 raft_consensus.cc:3096] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 2 FOLLOWER]: Advancing to term 3
W20261116 08:58:06.006353 20344 leader_election.cc:336] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9188aed91ff24b0ea766bf89f306b495 (127.18.102.61:36365): Network error: Client connection negotiation failed: client connection to 127.18.102.61:36365: connect: Connection refused (error 111)
I20261116 08:58:06.009955 20237 raft_consensus.cc:2490] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a356758e034e49e292899b82f51810cd in term 3.
I20261116 08:58:06.011430 20341 leader_election.cc:304] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a356758e034e49e292899b82f51810cd, d9a31f175a1a4361aace9c508bdb5291; no voters: 9188aed91ff24b0ea766bf89f306b495
I20261116 08:58:06.012383 20802 raft_consensus.cc:2826] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 3 FOLLOWER]: Leader election won for term 3
I20261116 08:58:06.013958 20802 raft_consensus.cc:705] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 3 LEADER]: Becoming Leader. State: Replica: a356758e034e49e292899b82f51810cd, State: Running, Role: LEADER
I20261116 08:58:06.014734 20802 consensus_queue.cc:238] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:06.019919 20805 sys_catalog.cc:455] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a356758e034e49e292899b82f51810cd. Latest consensus state: current_term: 3 leader_uuid: "a356758e034e49e292899b82f51810cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:06.020691 20805 sys_catalog.cc:458] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:06.022677 20237 raft_consensus.cc:1285] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 3 FOLLOWER]: Refusing update from remote peer a356758e034e49e292899b82f51810cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 3 index: 16. (index mismatch)
I20261116 08:58:06.024695 20802 consensus_queue.cc:1049] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.005s
I20261116 08:58:06.034850 20797 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a356758e034e49e292899b82f51810cd. Latest consensus state: current_term: 3 leader_uuid: "a356758e034e49e292899b82f51810cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:06.035781 20797 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: This master's current role is: FOLLOWER
W20261116 08:58:06.036041 20344 consensus_peers.cc:597] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd -> Peer 9188aed91ff24b0ea766bf89f306b495 (127.18.102.61:36365): Couldn't send request to peer 9188aed91ff24b0ea766bf89f306b495. Status: Network error: Client connection negotiation failed: client connection to 127.18.102.61:36365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:06.037647 20807 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:58:06.041097 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261116 08:58:06.042140 20797 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a356758e034e49e292899b82f51810cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:06.042788 20797 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:06.043572 20806 sys_catalog.cc:455] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a356758e034e49e292899b82f51810cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:06.044348 20806 sys_catalog.cc:458] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:06.045260 20807 catalog_manager.cc:694] Loaded metadata for table NewTableName [id=57a542ce297949569849767182f5b9d0]
W20261116 08:58:06.049185 20809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:06.053085 20807 tablet_loader.cc:96] loaded metadata for tablet 19ac1f3e0f6c4b37b015139ac43a746a (table NewTableName [id=57a542ce297949569849767182f5b9d0])
I20261116 08:58:06.054767 20807 tablet_loader.cc:96] loaded metadata for tablet 8663ba9cebca4dd0922589c3095d07f1 (table NewTableName [id=57a542ce297949569849767182f5b9d0])
I20261116 08:58:06.056252 20807 tablet_loader.cc:96] loaded metadata for tablet 86badd1d1e29431daebce9a9ae80f0c9 (table NewTableName [id=57a542ce297949569849767182f5b9d0])
I20261116 08:58:06.057830 20807 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:58:06.063764 20807 catalog_manager.cc:1310] Loaded cluster ID: f1b7bc8047cd4836a24134372a439417
I20261116 08:58:06.064024 20807 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
W20261116 08:58:06.067862 20810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:06.071707 20807 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:58:06.077128 20807 catalog_manager.cc:6232] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: Loaded TSK: 0
I20261116 08:58:06.078065 20807 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:58:06.085403 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:06.085534 20812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:06.086701 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:06.086973 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:06.087172 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819486087153 us; error 0 us; skew 500 ppm
I20261116 08:58:06.087796 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:06.090478 18840 webserver.cc:577] Webserver started at http://127.18.102.61:40741/ using document root <none> and password file <none>
I20261116 08:58:06.090915 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:06.091100 18840 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261116 08:58:06.098249 18840 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.001s
I20261116 08:58:06.101094 20817 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:06.101956 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261116 08:58:06.102274 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-1-root
uuid: "9188aed91ff24b0ea766bf89f306b495"
format_stamp: "Formatted at 2026-11-16 08:58:02 on dist-test-slave-qt0z"
I20261116 08:58:06.102588 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:06.133242 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:06.134654 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:06.186336 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.61:36365
I20261116 08:58:06.186447 20868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.61:36365 every 8 connection(s)
I20261116 08:58:06.196928 20598 heartbeater.cc:500] Master 127.18.102.60:32843 was elected leader, sending a full tablet report...
I20261116 08:58:06.210975 20869 sys_catalog.cc:263] Verifying existing consensus state
I20261116 08:58:06.217813 20520 heartbeater.cc:500] Master 127.18.102.60:32843 was elected leader, sending a full tablet report...
I20261116 08:58:06.219134 20869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: Bootstrap starting.
I20261116 08:58:06.237481 20676 heartbeater.cc:500] Master 127.18.102.60:32843 was elected leader, sending a full tablet report...
I20261116 08:58:06.253484 20599 heartbeater.cc:345] Connected to a master server at 127.18.102.61:36365
I20261116 08:58:06.274915 20356 catalog_manager.cc:6393] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: NewTableName [id=57a542ce297949569849767182f5b9d0] alter complete (version 1)
I20261116 08:58:06.295918 20521 heartbeater.cc:345] Connected to a master server at 127.18.102.61:36365
I20261116 08:58:06.296211 20677 heartbeater.cc:345] Connected to a master server at 127.18.102.61:36365
I20261116 08:58:06.347262 20869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=7 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261116 08:58:06.348502 20869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: Bootstrap complete.
I20261116 08:58:06.351873 20869 raft_consensus.cc:360] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:06.352622 20869 raft_consensus.cc:748] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9188aed91ff24b0ea766bf89f306b495, State: Initialized, Role: FOLLOWER
I20261116 08:58:06.353358 20869 consensus_queue.cc:261] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:06.355340 20877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:06.356194 20877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:06.356917 20869 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:06.357450 20844 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:58:06.407795 20877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a356758e034e49e292899b82f51810cd. Latest consensus state: current_term: 3 leader_uuid: "a356758e034e49e292899b82f51810cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:06.408134 20885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a356758e034e49e292899b82f51810cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:06.408588 20877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:06.408812 20885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:06.420552 20892 catalog_manager.cc:1310] Loaded cluster ID: f1b7bc8047cd4836a24134372a439417
I20261116 08:58:06.420804 20892 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: loading cluster ID for follower catalog manager: success
I20261116 08:58:06.422374 20806 sys_catalog.cc:455] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a356758e034e49e292899b82f51810cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:06.423130 20806 sys_catalog.cc:458] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:06.427412 20892 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: acquiring CA information for follower catalog manager: success
I20261116 08:58:06.431313 20892 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261116 08:58:06.477646 20833 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" instance_seqno: 1794819482661641) as {username='slave'} at 127.0.0.1:41160; Asking this server to re-register.
I20261116 08:58:06.479123 20599 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:06.482797 20833 ts_manager.cc:210] Registered new tserver with Master: e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:06.509552 20833 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" instance_seqno: 1794819482832204) as {username='slave'} at 127.0.0.1:41192; Asking this server to re-register.
I20261116 08:58:06.510488 20834 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d92186d919b741b4b8d69416caad3021" instance_seqno: 1794819482499512) as {username='slave'} at 127.0.0.1:41174; Asking this server to re-register.
I20261116 08:58:06.511322 20677 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:06.512415 20521 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:06.516137 20833 ts_manager.cc:210] Registered new tserver with Master: d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237)
I20261116 08:58:06.516350 20834 ts_manager.cc:210] Registered new tserver with Master: f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:06.522114 18840 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.318656153s
I20261116 08:58:06.551472 18840 master.cc:585] Master@127.18.102.60:32843 shutting down...
I20261116 08:58:06.588636 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 3 LEADER]: Raft consensus shutting down.
I20261116 08:58:06.589684 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 3 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:06.590209 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: stopping tablet replica
I20261116 08:58:06.615749 18840 master.cc:607] Master@127.18.102.60:32843 shutdown complete.
I20261116 08:58:06.728128 20885 raft_consensus.cc:499] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a356758e034e49e292899b82f51810cd)
I20261116 08:58:06.728575 20885 raft_consensus.cc:521] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:06.730516 20885 leader_election.cc:290] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9a31f175a1a4361aace9c508bdb5291 (127.18.102.62:32983), a356758e034e49e292899b82f51810cd (127.18.102.60:32843)
W20261116 08:58:06.740094 20818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.102.60:32843: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20261116 08:58:06.743549 20818 leader_election.cc:336] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a356758e034e49e292899b82f51810cd (127.18.102.60:32843): Network error: Client connection negotiation failed: client connection to 127.18.102.60:32843: connect: Connection refused (error 111)
I20261116 08:58:06.747360 20237 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9188aed91ff24b0ea766bf89f306b495" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "d9a31f175a1a4361aace9c508bdb5291" is_pre_election: true
I20261116 08:58:06.748636 20818 leader_election.cc:304] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9188aed91ff24b0ea766bf89f306b495; no voters: a356758e034e49e292899b82f51810cd, d9a31f175a1a4361aace9c508bdb5291
I20261116 08:58:06.749406 20885 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261116 08:58:06.772543 20897 raft_consensus.cc:499] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a356758e034e49e292899b82f51810cd)
I20261116 08:58:06.772996 20897 raft_consensus.cc:521] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:06.777886 20897 leader_election.cc:290] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9188aed91ff24b0ea766bf89f306b495 (127.18.102.61:36365), a356758e034e49e292899b82f51810cd (127.18.102.60:32843)
I20261116 08:58:06.789013 20844 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9a31f175a1a4361aace9c508bdb5291" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "9188aed91ff24b0ea766bf89f306b495" is_pre_election: true
I20261116 08:58:06.789896 20844 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9a31f175a1a4361aace9c508bdb5291 in term 3.
I20261116 08:58:06.791574 20214 leader_election.cc:304] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9188aed91ff24b0ea766bf89f306b495, d9a31f175a1a4361aace9c508bdb5291; no voters:
I20261116 08:58:06.792366 20897 raft_consensus.cc:2826] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261116 08:58:06.792685 20897 raft_consensus.cc:499] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a356758e034e49e292899b82f51810cd)
I20261116 08:58:06.792963 20897 raft_consensus.cc:3096] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 3 FOLLOWER]: Advancing to term 4
W20261116 08:58:06.792181 20211 leader_election.cc:336] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a356758e034e49e292899b82f51810cd (127.18.102.60:32843): Network error: Client connection negotiation failed: client connection to 127.18.102.60:32843: connect: Connection refused (error 111)
I20261116 08:58:06.797974 20897 raft_consensus.cc:521] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:06.799476 20897 leader_election.cc:290] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 4 election: Requested vote from peers 9188aed91ff24b0ea766bf89f306b495 (127.18.102.61:36365), a356758e034e49e292899b82f51810cd (127.18.102.60:32843)
I20261116 08:58:06.800158 20844 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9a31f175a1a4361aace9c508bdb5291" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "9188aed91ff24b0ea766bf89f306b495"
I20261116 08:58:06.800782 20844 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 3 FOLLOWER]: Advancing to term 4
W20261116 08:58:06.803390 20211 leader_election.cc:336] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a356758e034e49e292899b82f51810cd (127.18.102.60:32843): Network error: Client connection negotiation failed: client connection to 127.18.102.60:32843: connect: Connection refused (error 111)
I20261116 08:58:06.807724 20844 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a31f175a1a4361aace9c508bdb5291 in term 4.
I20261116 08:58:06.808828 20214 leader_election.cc:304] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9188aed91ff24b0ea766bf89f306b495, d9a31f175a1a4361aace9c508bdb5291; no voters: a356758e034e49e292899b82f51810cd
I20261116 08:58:06.809432 20897 raft_consensus.cc:2826] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 4 FOLLOWER]: Leader election won for term 4
I20261116 08:58:06.810472 20897 raft_consensus.cc:705] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 4 LEADER]: Becoming Leader. State: Replica: d9a31f175a1a4361aace9c508bdb5291, State: Running, Role: LEADER
I20261116 08:58:06.811259 20897 consensus_queue.cc:238] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 3.17, Last appended by leader: 17, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:06.815303 20902 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a31f175a1a4361aace9c508bdb5291. Latest consensus state: current_term: 4 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:06.816308 20902 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:06.817752 20904 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
W20261116 08:58:06.823346 20211 consensus_peers.cc:597] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 -> Peer a356758e034e49e292899b82f51810cd (127.18.102.60:32843): Couldn't send request to peer a356758e034e49e292899b82f51810cd. Status: Network error: Client connection negotiation failed: client connection to 127.18.102.60:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:06.824074 20844 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 4 FOLLOWER]: Refusing update from remote peer d9a31f175a1a4361aace9c508bdb5291: Log matching property violated. Preceding OpId in replica: term: 3 index: 17. Preceding OpId from leader: term: 4 index: 18. (index mismatch)
I20261116 08:58:06.825376 20904 catalog_manager.cc:694] Loaded metadata for table NewTableName [id=57a542ce297949569849767182f5b9d0]
I20261116 08:58:06.825575 20897 consensus_queue.cc:1049] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20261116 08:58:06.828459 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261116 08:58:06.831626 20885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a31f175a1a4361aace9c508bdb5291. Latest consensus state: current_term: 4 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:06.832345 20885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:06.832643 20904 tablet_loader.cc:96] loaded metadata for tablet 19ac1f3e0f6c4b37b015139ac43a746a (table NewTableName [id=57a542ce297949569849767182f5b9d0])
I20261116 08:58:06.834239 20904 tablet_loader.cc:96] loaded metadata for tablet 8663ba9cebca4dd0922589c3095d07f1 (table NewTableName [id=57a542ce297949569849767182f5b9d0])
I20261116 08:58:06.835765 20904 tablet_loader.cc:96] loaded metadata for tablet 86badd1d1e29431daebce9a9ae80f0c9 (table NewTableName [id=57a542ce297949569849767182f5b9d0])
I20261116 08:58:06.837674 20904 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:58:06.837602 20885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:06.838688 20885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:06.842964 20904 catalog_manager.cc:1310] Loaded cluster ID: f1b7bc8047cd4836a24134372a439417
I20261116 08:58:06.843545 20904 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:58:06.846270 20903 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
W20261116 08:58:06.846872 20905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:06.847077 20903 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:06.851421 20904 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:58:06.856573 20904 catalog_manager.cc:6232] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291: Loaded TSK: 0
I20261116 08:58:06.857580 20904 catalog_manager.cc:1565] Initializing in-progress tserver states...
W20261116 08:58:06.866250 20907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:06.874892 20909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:06.879441 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:06.880894 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:06.881184 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:06.881439 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819486881404 us; error 0 us; skew 500 ppm
I20261116 08:58:06.881965 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:06.884809 18840 webserver.cc:577] Webserver started at http://127.18.102.60:39727/ using document root <none> and password file <none>
I20261116 08:58:06.885293 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:06.885537 18840 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261116 08:58:06.890286 18840 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20261116 08:58:06.894531 20914 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:06.895469 18840 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20261116 08:58:06.895820 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-2-root
uuid: "a356758e034e49e292899b82f51810cd"
format_stamp: "Formatted at 2026-11-16 08:58:02 on dist-test-slave-qt0z"
I20261116 08:58:06.896078 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:06.916424 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:06.917512 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:06.972800 20679 heartbeater.cc:500] Master 127.18.102.62:32983 was elected leader, sending a full tablet report...
I20261116 08:58:06.980253 20600 heartbeater.cc:500] Master 127.18.102.62:32983 was elected leader, sending a full tablet report...
I20261116 08:58:06.987932 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.60:32843
I20261116 08:58:06.988044 20965 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.60:32843 every 8 connection(s)
I20261116 08:58:07.000329 20966 sys_catalog.cc:263] Verifying existing consensus state
I20261116 08:58:07.005111 20966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: Bootstrap starting.
I20261116 08:58:07.068145 20522 heartbeater.cc:500] Master 127.18.102.62:32983 was elected leader, sending a full tablet report...
I20261116 08:58:07.082949 20966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=7 ignored=0} mutations{seen=12 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261116 08:58:07.086438 20966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: Bootstrap complete.
I20261116 08:58:07.090142 20966 raft_consensus.cc:360] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:07.090981 20966 raft_consensus.cc:748] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: a356758e034e49e292899b82f51810cd, State: Initialized, Role: FOLLOWER
I20261116 08:58:07.091784 20966 consensus_queue.cc:261] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 3.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:07.100128 20973 sys_catalog.cc:455] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:07.104069 20973 sys_catalog.cc:458] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:07.103619 20966 sys_catalog.cc:565] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:07.112748 20941 raft_consensus.cc:3096] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 3 FOLLOWER]: Advancing to term 4
I20261116 08:58:07.127856 20520 heartbeater.cc:345] Connected to a master server at 127.18.102.60:32843
I20261116 08:58:07.139663 20676 heartbeater.cc:345] Connected to a master server at 127.18.102.60:32843
I20261116 08:58:07.143160 20598 heartbeater.cc:345] Connected to a master server at 127.18.102.60:32843
I20261116 08:58:07.166359 20981 sys_catalog.cc:455] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:07.167408 20981 sys_catalog.cc:458] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:07.167312 20973 sys_catalog.cc:455] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a31f175a1a4361aace9c508bdb5291. Latest consensus state: current_term: 4 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:07.168454 20973 sys_catalog.cc:458] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [sys.catalog]: This master's current role is: FOLLOWER
I20261116 08:58:07.180991 20903 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "d9a31f175a1a4361aace9c508bdb5291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } } }
I20261116 08:58:07.181520 20987 catalog_manager.cc:1310] Loaded cluster ID: f1b7bc8047cd4836a24134372a439417
I20261116 08:58:07.181600 20903 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:07.181907 20987 catalog_manager.cc:1603] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: loading cluster ID for follower catalog manager: success
I20261116 08:58:07.188874 20987 catalog_manager.cc:1625] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: acquiring CA information for follower catalog manager: success
I20261116 08:58:07.193676 20987 catalog_manager.cc:1653] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261116 08:58:07.234824 20931 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d92186d919b741b4b8d69416caad3021" instance_seqno: 1794819482499512) as {username='slave'} at 127.0.0.1:49590; Asking this server to re-register.
I20261116 08:58:07.236223 20520 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:07.239233 20931 ts_manager.cc:210] Registered new tserver with Master: d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237)
I20261116 08:58:07.250218 20931 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" instance_seqno: 1794819482661641) as {username='slave'} at 127.0.0.1:49610; Asking this server to re-register.
I20261116 08:58:07.250846 20930 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" instance_seqno: 1794819482832204) as {username='slave'} at 127.0.0.1:49600; Asking this server to re-register.
I20261116 08:58:07.252161 20598 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:07.252163 20676 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:07.255697 20930 ts_manager.cc:210] Registered new tserver with Master: e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:07.256064 20931 ts_manager.cc:210] Registered new tserver with Master: f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:07.257454 18840 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.261785832s
I20261116 08:58:24.018045 21017 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e950e37c4d2d425a9771e063fdaa2850)
I20261116 08:58:24.019979 21017 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.027638 21017 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:24.027263 20573 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:24.028865 21018 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d92186d919b741b4b8d69416caad3021)
I20261116 08:58:24.029428 21018 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.032198 21018 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:24.036934 21015 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d92186d919b741b4b8d69416caad3021)
I20261116 08:58:24.037479 21015 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.038686 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:24.040560 21015 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:24.041903 20492 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:24.057663 20573 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:24.058296 20573 raft_consensus.cc:2490] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5439f672b0b4560aa83c325f85cc6aa in term 2.
I20261116 08:58:24.060209 20616 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa; no voters: d92186d919b741b4b8d69416caad3021
I20261116 08:58:24.062992 20649 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:24.063968 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:24.064522 20650 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92186d919b741b4b8d69416caad3021 in term 3.
I20261116 08:58:24.065457 21018 raft_consensus.cc:499] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e950e37c4d2d425a9771e063fdaa2850)
I20261116 08:58:24.067073 21018 raft_consensus.cc:521] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.070308 21018 leader_election.cc:290] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:24.075556 21018 raft_consensus.cc:2826] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261116 08:58:24.072029 20459 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d92186d919b741b4b8d69416caad3021, f5439f672b0b4560aa83c325f85cc6aa; no voters: e950e37c4d2d425a9771e063fdaa2850
I20261116 08:58:24.075987 21018 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Starting leader election (detected failure of leader d92186d919b741b4b8d69416caad3021)
I20261116 08:58:24.073697 20538 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850; no voters: d92186d919b741b4b8d69416caad3021, f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:24.076372 21018 raft_consensus.cc:3096] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:58:24.076675 20995 raft_consensus.cc:2826] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261116 08:58:24.077008 20995 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e950e37c4d2d425a9771e063fdaa2850)
I20261116 08:58:24.077351 20995 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Advancing to term 4
I20261116 08:58:24.077445 21015 raft_consensus.cc:2771] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20261116 08:58:24.070981 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:24.078150 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "86badd1d1e29431daebce9a9ae80f0c9" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:24.074684 21019 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e950e37c4d2d425a9771e063fdaa2850)
I20261116 08:58:24.079360 21019 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.081405 21019 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:24.082538 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:24.081928 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:24.082266 20616 leader_election.cc:304] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5439f672b0b4560aa83c325f85cc6aa; no voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850
I20261116 08:58:24.073889 21020 raft_consensus.cc:499] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9a31f175a1a4361aace9c508bdb5291)
I20261116 08:58:24.084452 20616 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5439f672b0b4560aa83c325f85cc6aa; no voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850
I20261116 08:58:24.085680 21021 raft_consensus.cc:2771] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261116 08:58:24.085925 21019 raft_consensus.cc:2771] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261116 08:58:24.084271 21020 raft_consensus.cc:521] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a31f175a1a4361aace9c508bdb5291" member_type: VOTER last_known_addr { host: "127.18.102.62" port: 32983 } } peers { permanent_uuid: "9188aed91ff24b0ea766bf89f306b495" member_type: VOTER last_known_addr { host: "127.18.102.61" port: 36365 } } peers { permanent_uuid: "a356758e034e49e292899b82f51810cd" member_type: VOTER last_known_addr { host: "127.18.102.60" port: 32843 } }
I20261116 08:58:24.084897 20995 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.088575 20995 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 4 election: Requested vote from peers e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:24.089569 21018 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.090088 20651 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
I20261116 08:58:24.090318 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "d92186d919b741b4b8d69416caad3021" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850"
I20261116 08:58:24.091619 21018 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 3 election: Requested vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:24.091840 20459 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d92186d919b741b4b8d69416caad3021; no voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:24.092931 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850"
I20261116 08:58:24.093741 21020 leader_election.cc:290] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d9a31f175a1a4361aace9c508bdb5291 (127.18.102.62:32983), 9188aed91ff24b0ea766bf89f306b495 (127.18.102.61:36365)
I20261116 08:58:24.095497 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021"
I20261116 08:58:24.092528 20995 raft_consensus.cc:2771] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20261116 08:58:24.097888 20614 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5439f672b0b4560aa83c325f85cc6aa; no voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850
I20261116 08:58:24.099810 21018 raft_consensus.cc:2771] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20261116 08:58:24.123796 20237 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a356758e034e49e292899b82f51810cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 18 } } ignore_live_leader: false dest_uuid: "d9a31f175a1a4361aace9c508bdb5291" is_pre_election: true
I20261116 08:58:24.124774 20844 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a356758e034e49e292899b82f51810cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 18 } } ignore_live_leader: false dest_uuid: "9188aed91ff24b0ea766bf89f306b495" is_pre_election: true
I20261116 08:58:24.126051 20918 leader_election.cc:304] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a356758e034e49e292899b82f51810cd; no voters: 9188aed91ff24b0ea766bf89f306b495, d9a31f175a1a4361aace9c508bdb5291
I20261116 08:58:24.126587 21020 raft_consensus.cc:2771] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261116 08:58:24.148183 20651 raft_consensus.cc:1250] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Rejecting Update request from peer d92186d919b741b4b8d69416caad3021 for earlier term 2. Current term is 3. Ops: []
I20261116 08:58:24.149154 20995 consensus_queue.cc:1060] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20261116 08:58:24.149859 21017 raft_consensus.cc:3091] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 2 LEADER]: Stepping down as leader of term 2
I20261116 08:58:24.150149 21017 raft_consensus.cc:748] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d92186d919b741b4b8d69416caad3021, State: Running, Role: LEADER
I20261116 08:58:24.150797 21017 consensus_queue.cc:261] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.152021 21017 raft_consensus.cc:3096] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:58:24.156477 20494 raft_consensus.cc:1250] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 4 FOLLOWER]: Rejecting Update request from peer e950e37c4d2d425a9771e063fdaa2850 for earlier term 3. Current term is 4. Ops: []
I20261116 08:58:24.157801 21015 consensus_queue.cc:1060] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261116 08:58:24.159126 21008 raft_consensus.cc:3091] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 3 LEADER]: Stepping down as leader of term 3
I20261116 08:58:24.159433 21008 raft_consensus.cc:748] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Running, Role: LEADER
I20261116 08:58:24.160015 21008 consensus_queue.cc:261] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.160827 21008 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 3 FOLLOWER]: Advancing to term 4
I20261116 08:58:24.306421 21008 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d92186d919b741b4b8d69416caad3021)
I20261116 08:58:24.306825 21008 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.308231 21008 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:24.308956 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:24.309067 20651 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:24.309675 20494 raft_consensus.cc:2490] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 3.
I20261116 08:58:24.309777 20651 raft_consensus.cc:2415] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e950e37c4d2d425a9771e063fdaa2850 in current term 3: Already voted for candidate f5439f672b0b4560aa83c325f85cc6aa in this term.
I20261116 08:58:24.310745 20536 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850; no voters:
I20261116 08:58:24.311353 21008 raft_consensus.cc:2826] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261116 08:58:24.311628 21008 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d92186d919b741b4b8d69416caad3021)
I20261116 08:58:24.311878 21008 raft_consensus.cc:3096] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 2 FOLLOWER]: Advancing to term 3
I20261116 08:58:24.315801 21008 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.317066 21008 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 election: Requested vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:24.317788 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021"
I20261116 08:58:24.318044 21018 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:24.317991 20651 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
I20261116 08:58:24.318665 21018 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.319700 21008 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:24.319907 20651 raft_consensus.cc:2415] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e950e37c4d2d425a9771e063fdaa2850 in current term 3: Already voted for candidate f5439f672b0b4560aa83c325f85cc6aa in this term.
I20261116 08:58:24.320147 21008 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.320657 21018 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:24.322141 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:24.322482 21018 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e950e37c4d2d425a9771e063fdaa2850)
I20261116 08:58:24.322438 20492 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:24.322767 21008 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:24.322718 20572 raft_consensus.cc:2490] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5439f672b0b4560aa83c325f85cc6aa in term 3.
I20261116 08:58:24.323642 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa" is_pre_election: true
I20261116 08:58:24.324779 20616 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa; no voters:
I20261116 08:58:24.325537 21019 raft_consensus.cc:2826] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261116 08:58:24.325870 21019 raft_consensus.cc:499] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 08:58:24.325871 20494 raft_consensus.cc:2490] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 3.
I20261116 08:58:24.326450 21019 raft_consensus.cc:3096] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Advancing to term 4
I20261116 08:58:24.326973 20492 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:24.326465 21018 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.327787 20492 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 4.
I20261116 08:58:24.328002 20650 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 3.
I20261116 08:58:24.329105 20536 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850; no voters:
I20261116 08:58:24.330204 21008 raft_consensus.cc:2826] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20261116 08:58:24.330606 21008 raft_consensus.cc:499] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 08:58:24.330726 20536 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d92186d919b741b4b8d69416caad3021, e950e37c4d2d425a9771e063fdaa2850; no voters: f5439f672b0b4560aa83c325f85cc6aa
I20261116 08:58:24.331003 21008 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 4 FOLLOWER]: Advancing to term 5
I20261116 08:58:24.331736 21015 raft_consensus.cc:2826] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 3 FOLLOWER]: Leader election won for term 3
I20261116 08:58:24.332033 21018 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:24.332993 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021" is_pre_election: true
I20261116 08:58:24.333395 20572 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850" is_pre_election: true
I20261116 08:58:24.333755 20494 raft_consensus.cc:2415] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f5439f672b0b4560aa83c325f85cc6aa in current term 4: Already voted for candidate d92186d919b741b4b8d69416caad3021 in this term.
I20261116 08:58:24.335558 21015 raft_consensus.cc:705] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 3 LEADER]: Becoming Leader. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Running, Role: LEADER
I20261116 08:58:24.335387 21019 raft_consensus.cc:521] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.337842 20494 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021"
I20261116 08:58:24.338339 20494 raft_consensus.cc:3096] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Advancing to term 4
I20261116 08:58:24.338414 21008 raft_consensus.cc:521] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.339316 20573 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "19ac1f3e0f6c4b37b015139ac43a746a" candidate_uuid: "f5439f672b0b4560aa83c325f85cc6aa" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e950e37c4d2d425a9771e063fdaa2850"
I20261116 08:58:24.340317 21019 leader_election.cc:290] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 4 election: Requested vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591)
I20261116 08:58:24.341336 20572 raft_consensus.cc:2390] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f5439f672b0b4560aa83c325f85cc6aa for earlier term 4. Current term is 5.
I20261116 08:58:24.341279 20650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f5439f672b0b4560aa83c325f85cc6aa"
I20261116 08:58:24.341959 20650 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Advancing to term 5
I20261116 08:58:24.342015 20492 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8663ba9cebca4dd0922589c3095d07f1" candidate_uuid: "e950e37c4d2d425a9771e063fdaa2850" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d92186d919b741b4b8d69416caad3021"
I20261116 08:58:24.342614 20492 raft_consensus.cc:3096] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 4 FOLLOWER]: Advancing to term 5
I20261116 08:58:24.337040 21015 consensus_queue.cc:238] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.344238 20494 raft_consensus.cc:2490] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5439f672b0b4560aa83c325f85cc6aa in term 4.
I20261116 08:58:24.345216 20614 leader_election.cc:304] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d92186d919b741b4b8d69416caad3021, f5439f672b0b4560aa83c325f85cc6aa; no voters: e950e37c4d2d425a9771e063fdaa2850
I20261116 08:58:24.345969 21019 raft_consensus.cc:2826] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 4 FOLLOWER]: Leader election won for term 4
I20261116 08:58:24.347218 21019 raft_consensus.cc:705] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 4 LEADER]: Becoming Leader. State: Replica: f5439f672b0b4560aa83c325f85cc6aa, State: Running, Role: LEADER
I20261116 08:58:24.347891 20616 leader_election.cc:400] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 4 pre-election: Vote denied by peer e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591) with higher term. Message: Invalid argument: T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f5439f672b0b4560aa83c325f85cc6aa for earlier term 4. Current term is 5.
I20261116 08:58:24.348230 20616 leader_election.cc:403] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20261116 08:58:24.348114 21019 consensus_queue.cc:238] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.350729 20492 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 5.
I20261116 08:58:24.351114 20650 raft_consensus.cc:2490] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e950e37c4d2d425a9771e063fdaa2850 in term 5.
I20261116 08:58:24.351856 21018 raft_consensus.cc:2771] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591) with higher term. Message: Invalid argument: T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f5439f672b0b4560aa83c325f85cc6aa for earlier term 4. Current term is 5.
I20261116 08:58:24.352875 20538 leader_election.cc:304] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e950e37c4d2d425a9771e063fdaa2850, f5439f672b0b4560aa83c325f85cc6aa; no voters:
I20261116 08:58:24.354099 21026 raft_consensus.cc:2826] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 5 FOLLOWER]: Leader election won for term 5
I20261116 08:58:24.355772 21008 leader_election.cc:290] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [CANDIDATE]: Term 5 election: Requested vote from peers d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237), f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3:35323)
I20261116 08:58:24.357409 21026 raft_consensus.cc:705] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 5 LEADER]: Becoming Leader. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Running, Role: LEADER
I20261116 08:58:24.358531 21026 consensus_queue.cc:238] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.361399 20492 raft_consensus.cc:1250] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 4 FOLLOWER]: Rejecting Update request from peer e950e37c4d2d425a9771e063fdaa2850 for earlier term 3. Current term is 4. Ops: []
I20261116 08:58:24.366245 20223 catalog_manager.cc:5840] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 reported cstate change: term changed from 2 to 3, leader changed from d92186d919b741b4b8d69416caad3021 (127.18.102.1) to e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2). New cstate: current_term: 3 leader_uuid: "e950e37c4d2d425a9771e063fdaa2850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20261116 08:58:24.376035 21015 consensus_queue.cc:1060] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20261116 08:58:24.377431 21008 raft_consensus.cc:3091] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 3 LEADER]: Stepping down as leader of term 3
I20261116 08:58:24.377777 21008 raft_consensus.cc:748] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e950e37c4d2d425a9771e063fdaa2850, State: Running, Role: LEADER
I20261116 08:58:24.378561 21008 consensus_queue.cc:261] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } }
I20261116 08:58:24.379726 21008 raft_consensus.cc:3096] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 3 FOLLOWER]: Advancing to term 4
I20261116 08:58:24.393478 20650 raft_consensus.cc:1250] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 4 LEADER]: Rejecting Update request from peer e950e37c4d2d425a9771e063fdaa2850 for earlier term 3. Current term is 4. Ops: []
I20261116 08:58:24.394155 20492 raft_consensus.cc:1285] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 4 FOLLOWER]: Refusing update from remote peer f5439f672b0b4560aa83c325f85cc6aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20261116 08:58:24.394924 20572 raft_consensus.cc:1285] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 4 FOLLOWER]: Refusing update from remote peer f5439f672b0b4560aa83c325f85cc6aa: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20261116 08:58:24.395354 20572 pending_rounds.cc:85] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850: Aborting all ops after (but not including) 4
I20261116 08:58:24.395633 20572 pending_rounds.cc:107] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20261116 08:58:24.395967 20572 raft_consensus.cc:2917] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20261116 08:58:24.398377 21018 consensus_queue.cc:1049] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20261116 08:58:24.402912 20492 raft_consensus.cc:1285] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 5 FOLLOWER]: Refusing update from remote peer e950e37c4d2d425a9771e063fdaa2850: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20261116 08:58:24.404474 21018 consensus_queue.cc:1049] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20261116 08:58:24.405109 21029 consensus_queue.cc:1049] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20261116 08:58:24.408268 20651 raft_consensus.cc:1285] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 5 FOLLOWER]: Refusing update from remote peer e950e37c4d2d425a9771e063fdaa2850: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20261116 08:58:24.411276 21029 consensus_queue.cc:1049] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261116 08:58:24.476322 20224 catalog_manager.cc:5840] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa reported cstate change: term changed from 3 to 4, leader changed from e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2) to f5439f672b0b4560aa83c325f85cc6aa (127.18.102.3). New cstate: current_term: 4 leader_uuid: "f5439f672b0b4560aa83c325f85cc6aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } health_report { overall_health: HEALTHY } } }
I20261116 08:58:24.491969 20223 catalog_manager.cc:5840] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "e950e37c4d2d425a9771e063fdaa2850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92186d919b741b4b8d69416caad3021" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 37237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e950e37c4d2d425a9771e063fdaa2850" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5439f672b0b4560aa83c325f85cc6aa" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 35323 } health_report { overall_health: UNKNOWN } } }
W20261116 08:58:24.947562 20258 debug-util.cc:398] Leaking SignalData structure 0x7b080024faa0 after lost signal to thread 18843
W20261116 08:58:24.949002 20258 debug-util.cc:398] Leaking SignalData structure 0x7b080004fde0 after lost signal to thread 20519
W20261116 08:58:24.949798 20258 debug-util.cc:398] Leaking SignalData structure 0x7b080024f540 after lost signal to thread 20597
W20261116 08:58:24.950570 20258 debug-util.cc:398] Leaking SignalData structure 0x7b08000acda0 after lost signal to thread 20675
W20261116 08:58:24.952072 20258 debug-util.cc:398] Leaking SignalData structure 0x7b08001a8140 after lost signal to thread 20965
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:784: Failure
Expected equality of these values:
live_row_count
Which is: 67
table_info->GetMetrics()->live_row_count->value()
Which is: 134
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:768: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 784, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:896: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20261116 08:58:37.353844 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
W20261116 08:58:37.381949 20536 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 4 similar messages]
W20261116 08:58:37.386775 20536 consensus_peers.cc:597] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 -> Peer d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237): Couldn't send request to peer d92186d919b741b4b8d69416caad3021. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:37.391783 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:37.392501 18840 tablet_replica.cc:333] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021: stopping tablet replica
I20261116 08:58:37.393184 18840 raft_consensus.cc:2261] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 4 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:37.393678 18840 raft_consensus.cc:2290] T 19ac1f3e0f6c4b37b015139ac43a746a P d92186d919b741b4b8d69416caad3021 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:37.395766 18840 tablet_replica.cc:333] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021: stopping tablet replica
I20261116 08:58:37.396356 18840 raft_consensus.cc:2261] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 5 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:37.396770 18840 raft_consensus.cc:2290] T 8663ba9cebca4dd0922589c3095d07f1 P d92186d919b741b4b8d69416caad3021 [term 5 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:37.398779 18840 tablet_replica.cc:333] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021: stopping tablet replica
I20261116 08:58:37.399302 18840 raft_consensus.cc:2261] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:37.399730 18840 raft_consensus.cc:2290] T 86badd1d1e29431daebce9a9ae80f0c9 P d92186d919b741b4b8d69416caad3021 [term 3 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:37.421288 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
W20261116 08:58:37.429014 20536 consensus_peers.cc:597] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 -> Peer d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237): Couldn't send request to peer d92186d919b741b4b8d69416caad3021. Status: Network error: Client connection negotiation failed: client connection to 127.18.102.1:37237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:37.441247 18840 tablet_server.cc:179] TabletServer@127.18.102.2:0 shutting down...
W20261116 08:58:37.470742 20614 consensus_peers.cc:597] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa -> Peer d92186d919b741b4b8d69416caad3021 (127.18.102.1:37237): Couldn't send request to peer d92186d919b741b4b8d69416caad3021. Status: Network error: Client connection negotiation failed: client connection to 127.18.102.1:37237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:37.473860 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:37.474768 18840 tablet_replica.cc:333] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850: stopping tablet replica
I20261116 08:58:37.475556 18840 raft_consensus.cc:2261] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 4 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:37.476218 18840 raft_consensus.cc:2290] T 19ac1f3e0f6c4b37b015139ac43a746a P e950e37c4d2d425a9771e063fdaa2850 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:37.478772 18840 tablet_replica.cc:333] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850: stopping tablet replica
I20261116 08:58:37.479337 18840 raft_consensus.cc:2261] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 5 LEADER]: Raft consensus shutting down.
I20261116 08:58:37.480168 18840 raft_consensus.cc:2290] T 8663ba9cebca4dd0922589c3095d07f1 P e950e37c4d2d425a9771e063fdaa2850 [term 5 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:37.482405 18840 tablet_replica.cc:333] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850: stopping tablet replica
I20261116 08:58:37.482983 18840 raft_consensus.cc:2261] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 3 LEADER]: Raft consensus shutting down.
I20261116 08:58:37.483688 18840 raft_consensus.cc:2290] T 86badd1d1e29431daebce9a9ae80f0c9 P e950e37c4d2d425a9771e063fdaa2850 [term 3 FOLLOWER]: Raft consensus is shut down!
W20261116 08:58:37.487890 20616 consensus_peers.cc:597] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa -> Peer e950e37c4d2d425a9771e063fdaa2850 (127.18.102.2:37591): Couldn't send request to peer e950e37c4d2d425a9771e063fdaa2850. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:37.507601 18840 tablet_server.cc:196] TabletServer@127.18.102.2:0 shutdown complete.
I20261116 08:58:37.521029 18840 tablet_server.cc:179] TabletServer@127.18.102.3:0 shutting down...
I20261116 08:58:37.557578 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:37.558188 18840 tablet_replica.cc:333] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa: stopping tablet replica
I20261116 08:58:37.558780 18840 raft_consensus.cc:2261] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:37.559276 18840 raft_consensus.cc:2290] T 86badd1d1e29431daebce9a9ae80f0c9 P f5439f672b0b4560aa83c325f85cc6aa [term 3 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:37.561084 18840 tablet_replica.cc:333] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa: stopping tablet replica
I20261116 08:58:37.561497 18840 raft_consensus.cc:2261] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 5 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:37.561852 18840 raft_consensus.cc:2290] T 8663ba9cebca4dd0922589c3095d07f1 P f5439f672b0b4560aa83c325f85cc6aa [term 5 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:37.563609 18840 tablet_replica.cc:333] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa: stopping tablet replica
I20261116 08:58:37.564038 18840 raft_consensus.cc:2261] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 4 LEADER]: Raft consensus shutting down.
I20261116 08:58:37.564728 18840 raft_consensus.cc:2290] T 19ac1f3e0f6c4b37b015139ac43a746a P f5439f672b0b4560aa83c325f85cc6aa [term 4 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:37.586653 18840 tablet_server.cc:196] TabletServer@127.18.102.3:0 shutdown complete.
I20261116 08:58:37.602063 18840 master.cc:585] Master@127.18.102.62:32983 shutting down...
I20261116 08:58:37.623054 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 4 LEADER]: Raft consensus shutting down.
I20261116 08:58:37.624020 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:37.624405 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P d9a31f175a1a4361aace9c508bdb5291: stopping tablet replica
I20261116 08:58:37.676878 18840 master.cc:607] Master@127.18.102.62:32983 shutdown complete.
I20261116 08:58:37.693461 18840 master.cc:585] Master@127.18.102.61:36365 shutting down...
I20261116 08:58:37.716601 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 4 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:37.717178 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 9188aed91ff24b0ea766bf89f306b495 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:37.717510 18840 tablet_replica.cc:333] stopping tablet replica
I20261116 08:58:37.735109 18840 master.cc:607] Master@127.18.102.61:36365 shutdown complete.
I20261116 08:58:37.749644 18840 master.cc:585] Master@127.18.102.60:32843 shutting down...
I20261116 08:58:37.770462 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 4 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:37.770960 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P a356758e034e49e292899b82f51810cd [term 4 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:37.771318 18840 tablet_replica.cc:333] stopping tablet replica
I20261116 08:58:37.789800 18840 master.cc:607] Master@127.18.102.60:32843 shutdown complete.
I20261116 08:58:37.813861 18840 test_util.cc:182] -----------------------------------------------
I20261116 08:58:37.814038 18840 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1794819469408847-18840-0
[ FAILED ] TsTabletManagerITest.TestTableStats (35877 ms)
[ RUN ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20261116 08:58:37.820334 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:46027
I20261116 08:58:37.821440 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:37.827221 21078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:37.827284 21079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:37.829035 21081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:37.829988 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:37.831929 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:37.832167 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:37.832376 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819517832345 us; error 0 us; skew 500 ppm
I20261116 08:58:37.832870 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:37.835045 18840 webserver.cc:577] Webserver started at http://127.18.102.62:38475/ using document root <none> and password file <none>
I20261116 08:58:37.835579 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:37.835774 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:37.836035 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:37.837118 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "d1cd1752f2bc44809860f5aa3239143e"
format_stamp: "Formatted at 2026-11-16 08:58:37 on dist-test-slave-qt0z"
I20261116 08:58:37.841411 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20261116 08:58:37.844643 21086 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:37.845480 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20261116 08:58:37.845697 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "d1cd1752f2bc44809860f5aa3239143e"
format_stamp: "Formatted at 2026-11-16 08:58:37 on dist-test-slave-qt0z"
I20261116 08:58:37.845927 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:37.871649 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:37.872783 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:37.907907 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:46027
I20261116 08:58:37.908008 21137 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:46027 every 8 connection(s)
I20261116 08:58:37.911610 21138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:37.921625 21138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e: Bootstrap starting.
I20261116 08:58:37.926095 21138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:37.929859 21138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e: No bootstrap required, opened a new log
I20261116 08:58:37.931975 21138 raft_consensus.cc:360] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cd1752f2bc44809860f5aa3239143e" member_type: VOTER }
I20261116 08:58:37.932400 21138 raft_consensus.cc:386] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:37.932621 21138 raft_consensus.cc:748] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1cd1752f2bc44809860f5aa3239143e, State: Initialized, Role: FOLLOWER
I20261116 08:58:37.933233 21138 consensus_queue.cc:261] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cd1752f2bc44809860f5aa3239143e" member_type: VOTER }
I20261116 08:58:37.933676 21138 raft_consensus.cc:400] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:37.933911 21138 raft_consensus.cc:499] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:37.934172 21138 raft_consensus.cc:3096] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:37.938097 21138 raft_consensus.cc:521] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cd1752f2bc44809860f5aa3239143e" member_type: VOTER }
I20261116 08:58:37.938577 21138 leader_election.cc:304] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1cd1752f2bc44809860f5aa3239143e; no voters:
I20261116 08:58:37.939631 21138 leader_election.cc:290] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:37.940018 21141 raft_consensus.cc:2826] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:37.941296 21141 raft_consensus.cc:705] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [term 1 LEADER]: Becoming Leader. State: Replica: d1cd1752f2bc44809860f5aa3239143e, State: Running, Role: LEADER
I20261116 08:58:37.941836 21141 consensus_queue.cc:238] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cd1752f2bc44809860f5aa3239143e" member_type: VOTER }
I20261116 08:58:37.942348 21138 sys_catalog.cc:565] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:37.947789 21142 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1cd1752f2bc44809860f5aa3239143e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cd1752f2bc44809860f5aa3239143e" member_type: VOTER } }
I20261116 08:58:37.948383 21142 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:37.949932 21148 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:58:37.949841 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1cd1752f2bc44809860f5aa3239143e. Latest consensus state: current_term: 1 leader_uuid: "d1cd1752f2bc44809860f5aa3239143e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cd1752f2bc44809860f5aa3239143e" member_type: VOTER } }
I20261116 08:58:37.950567 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:37.954908 21148 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:58:37.959939 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:58:37.963903 21148 catalog_manager.cc:1398] Generated new cluster ID: d3f2c88e58c14b6084f6b36d78a2f3ca
I20261116 08:58:37.964118 21148 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:58:37.983762 21148 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:58:37.984874 21148 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:58:37.998443 21148 catalog_manager.cc:6221] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e: Generated new TSK 0
I20261116 08:58:37.998942 21148 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:58:38.025755 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:38.031437 21159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:38.032608 21160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:38.034003 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:38.033969 21162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:38.035248 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:38.035434 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:38.035552 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819518035541 us; error 0 us; skew 500 ppm
I20261116 08:58:38.035990 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:38.038126 18840 webserver.cc:577] Webserver started at http://127.18.102.1:37173/ using document root <none> and password file <none>
I20261116 08:58:38.038548 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:38.038693 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:38.038897 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:38.039968 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "af322bef80b74f748bfcfd3b50571991"
format_stamp: "Formatted at 2026-11-16 08:58:38 on dist-test-slave-qt0z"
I20261116 08:58:38.044405 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20261116 08:58:38.047293 21167 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:38.048084 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261116 08:58:38.048339 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "af322bef80b74f748bfcfd3b50571991"
format_stamp: "Formatted at 2026-11-16 08:58:38 on dist-test-slave-qt0z"
I20261116 08:58:38.048589 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:38.073537 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:38.074642 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:38.075996 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:38.078190 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:38.078382 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:38.078598 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:38.078779 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:38.115115 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:32793
I20261116 08:58:38.115285 21229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:32793 every 8 connection(s)
I20261116 08:58:38.119822 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:38.127419 21234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:38.128569 21235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:38.131572 21238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:38.131526 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:38.132697 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:38.132880 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:38.133111 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819518133089 us; error 0 us; skew 500 ppm
I20261116 08:58:38.133671 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:38.136155 18840 webserver.cc:577] Webserver started at http://127.18.102.2:39901/ using document root <none> and password file <none>
I20261116 08:58:38.136803 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:38.137033 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:38.137387 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:38.138470 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-1-root/instance:
uuid: "248bb86724e54f9ea7d12a12f55799e6"
format_stamp: "Formatted at 2026-11-16 08:58:38 on dist-test-slave-qt0z"
I20261116 08:58:38.138675 21230 heartbeater.cc:345] Connected to a master server at 127.18.102.62:46027
I20261116 08:58:38.139065 21230 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:38.139910 21230 heartbeater.cc:508] Master 127.18.102.62:46027 requested a full tablet report, sending...
I20261116 08:58:38.141996 21103 ts_manager.cc:210] Registered new tserver with Master: af322bef80b74f748bfcfd3b50571991 (127.18.102.1:32793)
I20261116 08:58:38.143466 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20261116 08:58:38.144143 21103 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57508
I20261116 08:58:38.147226 21242 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:38.147907 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261116 08:58:38.148165 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-1-root
uuid: "248bb86724e54f9ea7d12a12f55799e6"
format_stamp: "Formatted at 2026-11-16 08:58:38 on dist-test-slave-qt0z"
I20261116 08:58:38.148428 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:38.158813 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:38.159854 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:38.161128 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:38.162936 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:38.163134 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:38.163446 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:38.163614 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:38.199913 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.2:37823
I20261116 08:58:38.200006 21304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.2:37823 every 8 connection(s)
I20261116 08:58:38.204341 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:38.211854 21308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:38.212687 21309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:38.214756 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:38.215977 21311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:38.216100 21305 heartbeater.cc:345] Connected to a master server at 127.18.102.62:46027
I20261116 08:58:38.216475 21305 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:38.216882 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:38.217124 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:38.217203 21305 heartbeater.cc:508] Master 127.18.102.62:46027 requested a full tablet report, sending...
I20261116 08:58:38.217368 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819518217350 us; error 0 us; skew 500 ppm
I20261116 08:58:38.218094 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:38.219662 21103 ts_manager.cc:210] Registered new tserver with Master: 248bb86724e54f9ea7d12a12f55799e6 (127.18.102.2:37823)
I20261116 08:58:38.220635 18840 webserver.cc:577] Webserver started at http://127.18.102.3:39027/ using document root <none> and password file <none>
I20261116 08:58:38.221113 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:38.221155 21103 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57516
I20261116 08:58:38.221437 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:38.221782 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:38.223021 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-2-root/instance:
uuid: "66f181eddfb8426f8ac07f33e87b3f41"
format_stamp: "Formatted at 2026-11-16 08:58:38 on dist-test-slave-qt0z"
I20261116 08:58:38.227326 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20261116 08:58:38.230289 21316 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:38.230968 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261116 08:58:38.231259 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-2-root
uuid: "66f181eddfb8426f8ac07f33e87b3f41"
format_stamp: "Formatted at 2026-11-16 08:58:38 on dist-test-slave-qt0z"
I20261116 08:58:38.231565 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:38.249022 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:38.250317 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:38.251742 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:38.253734 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:38.253952 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:38.254177 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:38.254352 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:38.290638 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.3:38577
I20261116 08:58:38.290727 21378 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.3:38577 every 8 connection(s)
I20261116 08:58:38.303807 21379 heartbeater.cc:345] Connected to a master server at 127.18.102.62:46027
I20261116 08:58:38.304152 21379 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:38.304821 21379 heartbeater.cc:508] Master 127.18.102.62:46027 requested a full tablet report, sending...
I20261116 08:58:38.306692 21103 ts_manager.cc:210] Registered new tserver with Master: 66f181eddfb8426f8ac07f33e87b3f41 (127.18.102.3:38577)
I20261116 08:58:38.306826 18840 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013215946s
I20261116 08:58:38.308070 21103 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57528
I20261116 08:58:38.336592 21103 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20261116 08:58:38.338173 21103 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261116 08:58:38.384542 21344 tablet_service.cc:1500] Processing CreateTablet for tablet 57dca64951f64ccfbd8d543409e89280 (DEFAULT_TABLE table=test-table [id=6fb5ed8c51694d95b044b6d1fd6b8fc0]), partition=
I20261116 08:58:38.384943 21270 tablet_service.cc:1500] Processing CreateTablet for tablet 57dca64951f64ccfbd8d543409e89280 (DEFAULT_TABLE table=test-table [id=6fb5ed8c51694d95b044b6d1fd6b8fc0]), partition=
I20261116 08:58:38.385568 21344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57dca64951f64ccfbd8d543409e89280. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:38.385751 21195 tablet_service.cc:1500] Processing CreateTablet for tablet 57dca64951f64ccfbd8d543409e89280 (DEFAULT_TABLE table=test-table [id=6fb5ed8c51694d95b044b6d1fd6b8fc0]), partition=
I20261116 08:58:38.386020 21270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57dca64951f64ccfbd8d543409e89280. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:38.386812 21195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57dca64951f64ccfbd8d543409e89280. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:38.398380 21395 tablet_bootstrap.cc:492] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Bootstrap starting.
I20261116 08:58:38.402029 21396 tablet_bootstrap.cc:492] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6: Bootstrap starting.
I20261116 08:58:38.404055 21395 tablet_bootstrap.cc:654] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:38.408231 21397 tablet_bootstrap.cc:492] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991: Bootstrap starting.
I20261116 08:58:38.408865 21396 tablet_bootstrap.cc:654] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:38.409461 21395 tablet_bootstrap.cc:492] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: No bootstrap required, opened a new log
I20261116 08:58:38.409984 21395 ts_tablet_manager.cc:1449] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20261116 08:58:38.412968 21395 raft_consensus.cc:360] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } }
I20261116 08:58:38.413666 21395 raft_consensus.cc:386] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:38.414031 21396 tablet_bootstrap.cc:492] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6: No bootstrap required, opened a new log
I20261116 08:58:38.413970 21395 raft_consensus.cc:748] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f181eddfb8426f8ac07f33e87b3f41, State: Initialized, Role: FOLLOWER
I20261116 08:58:38.414561 21397 tablet_bootstrap.cc:654] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:38.414637 21396 ts_tablet_manager.cc:1449] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20261116 08:58:38.415113 21395 consensus_queue.cc:261] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } }
I20261116 08:58:38.418087 21396 raft_consensus.cc:360] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } }
I20261116 08:58:38.419014 21396 raft_consensus.cc:386] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:38.419510 21396 raft_consensus.cc:748] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248bb86724e54f9ea7d12a12f55799e6, State: Initialized, Role: FOLLOWER
I20261116 08:58:38.420539 21396 consensus_queue.cc:261] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } }
I20261116 08:58:38.421159 21395 ts_tablet_manager.cc:1480] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Time spent starting tablet: real 0.011s user 0.002s sys 0.007s
I20261116 08:58:38.421342 21397 tablet_bootstrap.cc:492] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991: No bootstrap required, opened a new log
W20261116 08:58:38.421496 21395 ts_tablet_manager.cc:1503] Injecting 50ms of latency into OpenTablet
I20261116 08:58:38.421841 21397 ts_tablet_manager.cc:1449] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20261116 08:58:38.422998 21305 heartbeater.cc:500] Master 127.18.102.62:46027 was elected leader, sending a full tablet report...
I20261116 08:58:38.423231 21379 heartbeater.cc:500] Master 127.18.102.62:46027 was elected leader, sending a full tablet report...
I20261116 08:58:38.423520 21396 ts_tablet_manager.cc:1480] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20261116 08:58:38.423760 21396 ts_tablet_manager.cc:1503] Injecting 50ms of latency into OpenTablet
I20261116 08:58:38.424266 21397 raft_consensus.cc:360] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } }
I20261116 08:58:38.424988 21397 raft_consensus.cc:386] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:38.425225 21397 raft_consensus.cc:748] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af322bef80b74f748bfcfd3b50571991, State: Initialized, Role: FOLLOWER
I20261116 08:58:38.425873 21397 consensus_queue.cc:261] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } }
I20261116 08:58:38.429769 21230 heartbeater.cc:500] Master 127.18.102.62:46027 was elected leader, sending a full tablet report...
I20261116 08:58:38.430859 21397 ts_tablet_manager.cc:1480] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991: Time spent starting tablet: real 0.009s user 0.000s sys 0.009s
W20261116 08:58:38.431207 21397 ts_tablet_manager.cc:1503] Injecting 50ms of latency into OpenTablet
W20261116 08:58:38.453635 21306 tablet.cc:2441] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261116 08:58:38.807658 21402 raft_consensus.cc:499] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:38.808100 21402 raft_consensus.cc:521] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } }
I20261116 08:58:38.809964 21402 leader_election.cc:290] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af322bef80b74f748bfcfd3b50571991 (127.18.102.1:32793), 66f181eddfb8426f8ac07f33e87b3f41 (127.18.102.3:38577)
I20261116 08:58:38.819936 21400 raft_consensus.cc:499] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:38.820560 21205 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "57dca64951f64ccfbd8d543409e89280" candidate_uuid: "248bb86724e54f9ea7d12a12f55799e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af322bef80b74f748bfcfd3b50571991" is_pre_election: true
I20261116 08:58:38.820483 21354 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "57dca64951f64ccfbd8d543409e89280" candidate_uuid: "248bb86724e54f9ea7d12a12f55799e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f181eddfb8426f8ac07f33e87b3f41" is_pre_election: true
I20261116 08:58:38.820384 21400 raft_consensus.cc:521] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } }
I20261116 08:58:38.821302 21205 raft_consensus.cc:2490] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248bb86724e54f9ea7d12a12f55799e6 in term 0.
I20261116 08:58:38.821735 21354 raft_consensus.cc:2490] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248bb86724e54f9ea7d12a12f55799e6 in term 0.
I20261116 08:58:38.822634 21244 leader_election.cc:304] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248bb86724e54f9ea7d12a12f55799e6, af322bef80b74f748bfcfd3b50571991; no voters:
I20261116 08:58:38.823387 21400 leader_election.cc:290] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af322bef80b74f748bfcfd3b50571991 (127.18.102.1:32793), 248bb86724e54f9ea7d12a12f55799e6 (127.18.102.2:37823)
I20261116 08:58:38.823530 21402 raft_consensus.cc:2826] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 08:58:38.823894 21402 raft_consensus.cc:499] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 08:58:38.824251 21402 raft_consensus.cc:3096] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:38.831784 21402 raft_consensus.cc:521] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } }
I20261116 08:58:38.834564 21205 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "57dca64951f64ccfbd8d543409e89280" candidate_uuid: "248bb86724e54f9ea7d12a12f55799e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af322bef80b74f748bfcfd3b50571991"
I20261116 08:58:38.834764 21402 leader_election.cc:290] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [CANDIDATE]: Term 1 election: Requested vote from peers af322bef80b74f748bfcfd3b50571991 (127.18.102.1:32793), 66f181eddfb8426f8ac07f33e87b3f41 (127.18.102.3:38577)
I20261116 08:58:38.835120 21205 raft_consensus.cc:3096] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:38.836608 21204 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "57dca64951f64ccfbd8d543409e89280" candidate_uuid: "66f181eddfb8426f8ac07f33e87b3f41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af322bef80b74f748bfcfd3b50571991" is_pre_election: true
I20261116 08:58:38.837291 21354 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "57dca64951f64ccfbd8d543409e89280" candidate_uuid: "248bb86724e54f9ea7d12a12f55799e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f181eddfb8426f8ac07f33e87b3f41"
I20261116 08:58:38.837847 21354 raft_consensus.cc:3096] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:38.843041 21205 raft_consensus.cc:2490] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248bb86724e54f9ea7d12a12f55799e6 in term 1.
I20261116 08:58:38.844183 21244 leader_election.cc:304] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248bb86724e54f9ea7d12a12f55799e6, af322bef80b74f748bfcfd3b50571991; no voters:
I20261116 08:58:38.844420 21280 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "57dca64951f64ccfbd8d543409e89280" candidate_uuid: "66f181eddfb8426f8ac07f33e87b3f41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "248bb86724e54f9ea7d12a12f55799e6" is_pre_election: true
I20261116 08:58:38.844615 21354 raft_consensus.cc:2490] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248bb86724e54f9ea7d12a12f55799e6 in term 1.
I20261116 08:58:38.845175 21280 raft_consensus.cc:2415] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66f181eddfb8426f8ac07f33e87b3f41 in current term 1: Already voted for candidate 248bb86724e54f9ea7d12a12f55799e6 in this term.
I20261116 08:58:38.845906 21402 raft_consensus.cc:2826] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:38.846318 21317 leader_election.cc:304] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f181eddfb8426f8ac07f33e87b3f41; no voters: 248bb86724e54f9ea7d12a12f55799e6, af322bef80b74f748bfcfd3b50571991
I20261116 08:58:38.846988 21400 raft_consensus.cc:2771] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261116 08:58:38.847023 21402 raft_consensus.cc:705] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 1 LEADER]: Becoming Leader. State: Replica: 248bb86724e54f9ea7d12a12f55799e6, State: Running, Role: LEADER
I20261116 08:58:38.848027 21402 consensus_queue.cc:238] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } }
I20261116 08:58:38.854454 21408 consensus_queue.cc:1049] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261116 08:58:38.855989 21103 catalog_manager.cc:5840] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 248bb86724e54f9ea7d12a12f55799e6 (127.18.102.2). New cstate: current_term: 1 leader_uuid: "248bb86724e54f9ea7d12a12f55799e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } health_report { overall_health: HEALTHY } } }
I20261116 08:58:38.861047 21402 consensus_queue.cc:1049] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 08:58:38.982923 21344 tablet_service.cc:1547] Processing DeleteTablet for tablet 57dca64951f64ccfbd8d543409e89280 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51162
I20261116 08:58:38.984326 21420 tablet_replica.cc:333] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: stopping tablet replica
I20261116 08:58:38.984941 21420 raft_consensus.cc:2261] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:38.985427 21420 raft_consensus.cc:2290] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:38.987780 21420 ts_tablet_manager.cc:1962] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20261116 08:58:38.997011 21420 ts_tablet_manager.cc:1975] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20261116 08:58:38.997344 21420 log.cc:1199] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Deleting WAL directory at /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-2-root/wals/57dca64951f64ccfbd8d543409e89280
I20261116 08:58:39.000918 21421 ts_tablet_manager.cc:979] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Initiating tablet copy from peer 248bb86724e54f9ea7d12a12f55799e6 (127.18.102.2:37823)
I20261116 08:58:39.002210 21421 tablet_copy_client.cc:323] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: tablet copy: Beginning tablet copy session from remote peer at address 127.18.102.2:37823
I20261116 08:58:39.003698 21290 tablet_copy_service.cc:140] P 248bb86724e54f9ea7d12a12f55799e6: Received BeginTabletCopySession request for tablet 57dca64951f64ccfbd8d543409e89280 from peer 66f181eddfb8426f8ac07f33e87b3f41 ({username='slave'} at 127.0.0.1:52552)
I20261116 08:58:39.004086 21290 tablet_copy_service.cc:161] P 248bb86724e54f9ea7d12a12f55799e6: Beginning new tablet copy session on tablet 57dca64951f64ccfbd8d543409e89280 from peer 66f181eddfb8426f8ac07f33e87b3f41 at {username='slave'} at 127.0.0.1:52552: session id = 66f181eddfb8426f8ac07f33e87b3f41-57dca64951f64ccfbd8d543409e89280
I20261116 08:58:39.008370 21290 tablet_copy_source_session.cc:215] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6: Tablet Copy: opened 0 blocks and 1 log segments
I20261116 08:58:39.010509 21421 ts_tablet_manager.cc:1962] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Deleting tablet data with delete state TABLET_DATA_COPYING
I20261116 08:58:39.019558 21421 ts_tablet_manager.cc:1975] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20261116 08:58:39.020152 21421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57dca64951f64ccfbd8d543409e89280. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:39.024874 21421 tablet_copy_client.cc:806] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: tablet copy: Starting download of 0 data blocks...
I20261116 08:58:39.025293 21421 tablet_copy_client.cc:670] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: tablet copy: Starting download of 1 WAL segments...
I20261116 08:58:39.026978 21102 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57550:
table { table_id: "6fb5ed8c51694d95b044b6d1fd6b8fc0" table_name: "test-table" }
I20261116 08:58:39.027683 21102 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57550:
table { table_id: "6fb5ed8c51694d95b044b6d1fd6b8fc0" table_name: "test-table" }
I20261116 08:58:39.029172 21421 tablet_copy_client.cc:538] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261116 08:58:39.035872 21421 tablet_bootstrap.cc:492] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Bootstrap starting.
I20261116 08:58:39.037954 21102 catalog_manager.cc:6135] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e: Sending DeleteTablet for 3 replicas of tablet 57dca64951f64ccfbd8d543409e89280
I20261116 08:58:39.039999 21195 tablet_service.cc:1547] Processing DeleteTablet for tablet 57dca64951f64ccfbd8d543409e89280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-16 08:58:39 UTC) from {username='slave'} at 127.0.0.1:35954
I20261116 08:58:39.040535 21344 tablet_service.cc:1547] Processing DeleteTablet for tablet 57dca64951f64ccfbd8d543409e89280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-16 08:58:39 UTC) from {username='slave'} at 127.0.0.1:51146
I20261116 08:58:39.040910 21270 tablet_service.cc:1547] Processing DeleteTablet for tablet 57dca64951f64ccfbd8d543409e89280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-16 08:58:39 UTC) from {username='slave'} at 127.0.0.1:52550
W20261116 08:58:39.043007 21089 catalog_manager.cc:5045] TS 66f181eddfb8426f8ac07f33e87b3f41 (127.18.102.3:38577): delete failed for tablet 57dca64951f64ccfbd8d543409e89280 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57dca64951f64ccfbd8d543409e89280 already in progress: copying tablet
I20261116 08:58:39.045506 21424 tablet_replica.cc:333] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6: stopping tablet replica
I20261116 08:58:39.046895 21423 tablet_replica.cc:333] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991: stopping tablet replica
I20261116 08:58:39.047053 21424 raft_consensus.cc:2261] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:39.047752 21423 raft_consensus.cc:2261] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:39.048134 21424 raft_consensus.cc:2290] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:39.048409 21423 raft_consensus.cc:2290] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:39.051282 21424 ts_tablet_manager.cc:1962] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261116 08:58:39.051282 21423 ts_tablet_manager.cc:1962] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261116 08:58:39.054244 21421 tablet_bootstrap.cc:492] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261116 08:58:39.055091 21421 tablet_bootstrap.cc:492] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Bootstrap complete.
I20261116 08:58:39.055744 21421 ts_tablet_manager.cc:1449] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.005s
I20261116 08:58:39.058636 21421 raft_consensus.cc:360] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } }
I20261116 08:58:39.059341 21421 raft_consensus.cc:748] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f181eddfb8426f8ac07f33e87b3f41, State: Initialized, Role: FOLLOWER
I20261116 08:58:39.060133 21421 consensus_queue.cc:261] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af322bef80b74f748bfcfd3b50571991" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 32793 } } peers { permanent_uuid: "66f181eddfb8426f8ac07f33e87b3f41" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 38577 } } peers { permanent_uuid: "248bb86724e54f9ea7d12a12f55799e6" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 37823 } }
I20261116 08:58:39.062094 21421 ts_tablet_manager.cc:1480] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20261116 08:58:39.062397 21421 ts_tablet_manager.cc:1503] Injecting 50ms of latency into OpenTablet
I20261116 08:58:39.062415 21424 ts_tablet_manager.cc:1975] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261116 08:58:39.063040 21424 log.cc:1199] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6: Deleting WAL directory at /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-1-root/wals/57dca64951f64ccfbd8d543409e89280
I20261116 08:58:39.063817 21424 ts_tablet_manager.cc:1996] T 57dca64951f64ccfbd8d543409e89280 P 248bb86724e54f9ea7d12a12f55799e6: Deleting consensus metadata
I20261116 08:58:39.065394 21087 catalog_manager.cc:5065] TS 248bb86724e54f9ea7d12a12f55799e6 (127.18.102.2:37823): tablet 57dca64951f64ccfbd8d543409e89280 (table test-table [id=6fb5ed8c51694d95b044b6d1fd6b8fc0]) successfully deleted
I20261116 08:58:39.065627 21423 ts_tablet_manager.cc:1975] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261116 08:58:39.066031 21423 log.cc:1199] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991: Deleting WAL directory at /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-0-root/wals/57dca64951f64ccfbd8d543409e89280
I20261116 08:58:39.066747 21423 ts_tablet_manager.cc:1996] T 57dca64951f64ccfbd8d543409e89280 P af322bef80b74f748bfcfd3b50571991: Deleting consensus metadata
I20261116 08:58:39.067562 21344 tablet_service.cc:1547] Processing DeleteTablet for tablet 57dca64951f64ccfbd8d543409e89280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-16 08:58:39 UTC) from {username='slave'} at 127.0.0.1:51146
W20261116 08:58:39.069119 21089 catalog_manager.cc:5045] TS 66f181eddfb8426f8ac07f33e87b3f41 (127.18.102.3:38577): delete failed for tablet 57dca64951f64ccfbd8d543409e89280 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57dca64951f64ccfbd8d543409e89280 already in progress: copying tablet
I20261116 08:58:39.069562 21088 catalog_manager.cc:5065] TS af322bef80b74f748bfcfd3b50571991 (127.18.102.1:32793): tablet 57dca64951f64ccfbd8d543409e89280 (table test-table [id=6fb5ed8c51694d95b044b6d1fd6b8fc0]) successfully deleted
I20261116 08:58:39.114686 21290 tablet_copy_service.cc:342] P 248bb86724e54f9ea7d12a12f55799e6: Request end of tablet copy session 66f181eddfb8426f8ac07f33e87b3f41-57dca64951f64ccfbd8d543409e89280 received from {username='slave'} at 127.0.0.1:52552
I20261116 08:58:39.115207 21290 tablet_copy_service.cc:434] P 248bb86724e54f9ea7d12a12f55799e6: ending tablet copy session 66f181eddfb8426f8ac07f33e87b3f41-57dca64951f64ccfbd8d543409e89280 on tablet 57dca64951f64ccfbd8d543409e89280 with peer 66f181eddfb8426f8ac07f33e87b3f41
I20261116 08:58:39.118244 21344 tablet_service.cc:1547] Processing DeleteTablet for tablet 57dca64951f64ccfbd8d543409e89280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-16 08:58:39 UTC) from {username='slave'} at 127.0.0.1:51146
I20261116 08:58:39.119031 21420 tablet_replica.cc:333] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: stopping tablet replica
I20261116 08:58:39.119621 21420 raft_consensus.cc:2261] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:39.120018 21420 raft_consensus.cc:2290] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:39.122188 21420 ts_tablet_manager.cc:1962] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261116 08:58:39.132634 21420 ts_tablet_manager.cc:1975] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261116 08:58:39.132927 21420 log.cc:1199] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Deleting WAL directory at /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1794819469408847-18840-0/minicluster-data/ts-2-root/wals/57dca64951f64ccfbd8d543409e89280
I20261116 08:58:39.133551 21420 ts_tablet_manager.cc:1996] T 57dca64951f64ccfbd8d543409e89280 P 66f181eddfb8426f8ac07f33e87b3f41: Deleting consensus metadata
I20261116 08:58:39.135634 21089 catalog_manager.cc:5065] TS 66f181eddfb8426f8ac07f33e87b3f41 (127.18.102.3:38577): tablet 57dca64951f64ccfbd8d543409e89280 (table test-table [id=6fb5ed8c51694d95b044b6d1fd6b8fc0]) successfully deleted
I20261116 08:58:39.179692 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
I20261116 08:58:39.197309 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:39.213521 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
I20261116 08:58:39.221526 18840 tablet_server.cc:179] TabletServer@127.18.102.2:0 shutting down...
I20261116 08:58:39.236277 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:39.253204 18840 tablet_server.cc:196] TabletServer@127.18.102.2:0 shutdown complete.
I20261116 08:58:39.261495 18840 tablet_server.cc:179] TabletServer@127.18.102.3:0 shutting down...
I20261116 08:58:39.277495 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:39.294299 18840 tablet_server.cc:196] TabletServer@127.18.102.3:0 shutdown complete.
I20261116 08:58:39.303011 18840 master.cc:585] Master@127.18.102.62:46027 shutting down...
I20261116 08:58:39.317153 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:39.317606 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:39.317853 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P d1cd1752f2bc44809860f5aa3239143e: stopping tablet replica
I20261116 08:58:39.336788 18840 master.cc:607] Master@127.18.102.62:46027 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1540 ms)
[----------] 4 tests from TsTabletManagerITest (44474 ms total)
[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20261116 08:58:39.360836 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:33601
I20261116 08:58:39.361861 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:39.366297 21430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:39.367605 21431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:39.369176 21433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:39.369707 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:39.370365 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:39.370550 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:39.370710 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819519370679 us; error 0 us; skew 500 ppm
I20261116 08:58:39.371318 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:39.373502 18840 webserver.cc:577] Webserver started at http://127.18.102.62:37849/ using document root <none> and password file <none>
I20261116 08:58:39.373983 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:39.374166 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:39.374469 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:39.375558 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "7fd313ca9e494eb2a2321e222168f93a"
format_stamp: "Formatted at 2026-11-16 08:58:39 on dist-test-slave-qt0z"
I20261116 08:58:39.379590 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20261116 08:58:39.382490 21438 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:39.383204 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261116 08:58:39.383502 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "7fd313ca9e494eb2a2321e222168f93a"
format_stamp: "Formatted at 2026-11-16 08:58:39 on dist-test-slave-qt0z"
I20261116 08:58:39.383767 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:39.409386 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:39.410332 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:39.442706 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:33601
I20261116 08:58:39.442799 21489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:33601 every 8 connection(s)
I20261116 08:58:39.446310 21490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:39.455721 21490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a: Bootstrap starting.
I20261116 08:58:39.460158 21490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:39.463652 21490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a: No bootstrap required, opened a new log
I20261116 08:58:39.465515 21490 raft_consensus.cc:360] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd313ca9e494eb2a2321e222168f93a" member_type: VOTER }
I20261116 08:58:39.465875 21490 raft_consensus.cc:386] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:39.466046 21490 raft_consensus.cc:748] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd313ca9e494eb2a2321e222168f93a, State: Initialized, Role: FOLLOWER
I20261116 08:58:39.466601 21490 consensus_queue.cc:261] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd313ca9e494eb2a2321e222168f93a" member_type: VOTER }
I20261116 08:58:39.467018 21490 raft_consensus.cc:400] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:39.467211 21490 raft_consensus.cc:499] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:39.467423 21490 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:39.471204 21490 raft_consensus.cc:521] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd313ca9e494eb2a2321e222168f93a" member_type: VOTER }
I20261116 08:58:39.471635 21490 leader_election.cc:304] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fd313ca9e494eb2a2321e222168f93a; no voters:
I20261116 08:58:39.472620 21490 leader_election.cc:290] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:39.472926 21493 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:39.474210 21493 raft_consensus.cc:705] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [term 1 LEADER]: Becoming Leader. State: Replica: 7fd313ca9e494eb2a2321e222168f93a, State: Running, Role: LEADER
I20261116 08:58:39.474852 21493 consensus_queue.cc:238] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd313ca9e494eb2a2321e222168f93a" member_type: VOTER }
I20261116 08:58:39.475447 21490 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:39.477600 21495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fd313ca9e494eb2a2321e222168f93a. Latest consensus state: current_term: 1 leader_uuid: "7fd313ca9e494eb2a2321e222168f93a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd313ca9e494eb2a2321e222168f93a" member_type: VOTER } }
I20261116 08:58:39.477674 21494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fd313ca9e494eb2a2321e222168f93a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd313ca9e494eb2a2321e222168f93a" member_type: VOTER } }
I20261116 08:58:39.478281 21495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:39.478327 21494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:39.481842 21499 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:58:39.489326 21499 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:58:39.490248 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:58:39.497521 21499 catalog_manager.cc:1398] Generated new cluster ID: fe31354615354aaab2f6838610d76a11
I20261116 08:58:39.497767 21499 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:58:39.517798 21499 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:58:39.518957 21499 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:58:39.531946 21499 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a: Generated new TSK 0
I20261116 08:58:39.532447 21499 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:58:39.556541 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:39.562054 21511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:39.563454 21512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:39.564801 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:39.564822 21514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:39.565799 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:39.566015 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:39.566179 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819519566163 us; error 0 us; skew 500 ppm
I20261116 08:58:39.566671 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:39.568742 18840 webserver.cc:577] Webserver started at http://127.18.102.1:38849/ using document root <none> and password file <none>
I20261116 08:58:39.569231 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:39.569443 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:39.569713 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:39.570731 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "e6a10295c56d4db093049f2bf08fcea9"
format_stamp: "Formatted at 2026-11-16 08:58:39 on dist-test-slave-qt0z"
I20261116 08:58:39.575075 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20261116 08:58:39.578378 21519 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:39.579264 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261116 08:58:39.579501 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "e6a10295c56d4db093049f2bf08fcea9"
format_stamp: "Formatted at 2026-11-16 08:58:39 on dist-test-slave-qt0z"
I20261116 08:58:39.579727 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:39.600147 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:39.601087 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:39.605654 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:39.605988 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20261116 08:58:39.606309 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:39.606535 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:39.653705 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:45095
I20261116 08:58:39.653820 21589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:45095 every 8 connection(s)
I20261116 08:58:39.658999 21590 heartbeater.cc:345] Connected to a master server at 127.18.102.62:33601
I20261116 08:58:39.659593 21590 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:39.660440 21590 heartbeater.cc:508] Master 127.18.102.62:33601 requested a full tablet report, sending...
I20261116 08:58:39.662429 21455 ts_manager.cc:210] Registered new tserver with Master: e6a10295c56d4db093049f2bf08fcea9 (127.18.102.1:45095)
I20261116 08:58:39.663259 18840 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.00607604s
I20261116 08:58:39.664527 21455 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32832
I20261116 08:58:39.692896 21555 tablet_service.cc:1500] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261116 08:58:39.694098 21555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:39.704385 21600 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: Bootstrap starting.
I20261116 08:58:39.708477 21600 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:39.712025 21600 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: No bootstrap required, opened a new log
I20261116 08:58:39.712424 21600 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20261116 08:58:39.714222 21600 raft_consensus.cc:360] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "e6a10295c56d4db093049f2bf08fcea9" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45095 } }
I20261116 08:58:39.714655 21600 raft_consensus.cc:386] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:39.714872 21600 raft_consensus.cc:748] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a10295c56d4db093049f2bf08fcea9, State: Initialized, Role: FOLLOWER
I20261116 08:58:39.715397 21600 consensus_queue.cc:261] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e6a10295c56d4db093049f2bf08fcea9" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45095 } }
I20261116 08:58:39.715835 21600 raft_consensus.cc:400] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:39.716070 21600 raft_consensus.cc:499] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:39.716332 21600 raft_consensus.cc:3096] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:39.720418 21600 raft_consensus.cc:521] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "e6a10295c56d4db093049f2bf08fcea9" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45095 } }
I20261116 08:58:39.720906 21600 leader_election.cc:304] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6a10295c56d4db093049f2bf08fcea9; no voters:
I20261116 08:58:39.721943 21600 leader_election.cc:290] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:39.722257 21602 raft_consensus.cc:2826] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:39.723555 21602 raft_consensus.cc:705] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 1 LEADER]: Becoming Leader. State: Replica: e6a10295c56d4db093049f2bf08fcea9, State: Running, Role: LEADER
I20261116 08:58:39.723757 21590 heartbeater.cc:500] Master 127.18.102.62:33601 was elected leader, sending a full tablet report...
I20261116 08:58:39.724321 21600 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20261116 08:58:39.724337 21602 consensus_queue.cc:238] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e6a10295c56d4db093049f2bf08fcea9" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45095 } }
I20261116 08:58:39.728149 21604 tablet_replica.cc:442] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: TxnStatusTablet state changed. Reason: New leader e6a10295c56d4db093049f2bf08fcea9. Latest consensus state: current_term: 1 leader_uuid: "e6a10295c56d4db093049f2bf08fcea9" committed_config { opid_index: -1 peers { permanent_uuid: "e6a10295c56d4db093049f2bf08fcea9" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45095 } } }
I20261116 08:58:39.728752 21604 tablet_replica.cc:445] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: This TxnStatusTablet replica's current role is: LEADER
I20261116 08:58:39.730197 21603 tablet_replica.cc:442] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6a10295c56d4db093049f2bf08fcea9" committed_config { opid_index: -1 peers { permanent_uuid: "e6a10295c56d4db093049f2bf08fcea9" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45095 } } }
I20261116 08:58:39.730700 21603 tablet_replica.cc:445] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: This TxnStatusTablet replica's current role is: LEADER
I20261116 08:58:39.730422 21607 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261116 08:58:39.731247 21607 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20261116 08:58:39.732295 21455 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
W20261116 08:58:39.736151 21455 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261116 08:58:39.768749 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
I20261116 08:58:39.788689 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:39.789918 18840 tablet_replica.cc:333] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: stopping tablet replica
I20261116 08:58:39.790663 18840 raft_consensus.cc:2261] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:39.791469 18840 raft_consensus.cc:2290] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:39.836786 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
I20261116 08:58:39.862613 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:39.871603 21612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:39.871582 21611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:39.874511 21614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:39.875329 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:39.876641 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:39.876919 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:39.877142 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819519877125 us; error 0 us; skew 500 ppm
I20261116 08:58:39.877862 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:39.881157 18840 webserver.cc:577] Webserver started at http://127.18.102.1:38849/ using document root <none> and password file <none>
I20261116 08:58:39.881796 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:39.882073 18840 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261116 08:58:39.887349 18840 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20261116 08:58:39.890991 21619 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:39.891813 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261116 08:58:39.892150 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "e6a10295c56d4db093049f2bf08fcea9"
format_stamp: "Formatted at 2026-11-16 08:58:39 on dist-test-slave-qt0z"
I20261116 08:58:39.892712 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:39.921319 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:39.922616 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:39.930337 21628 ts_tablet_manager.cc:594] Loading tablet metadata (0/1 complete)
I20261116 08:58:39.940591 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261116 08:58:39.940891 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.012s user 0.001s sys 0.001s
I20261116 08:58:39.941191 18840 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261116 08:58:39.949396 21628 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: Bootstrap starting.
I20261116 08:58:39.952258 18840 ts_tablet_manager.cc:662] Registered 1 tablets
I20261116 08:58:39.952649 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.011s user 0.004s sys 0.004s
I20261116 08:58:39.992368 21628 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261116 08:58:39.993378 21628 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: Bootstrap complete.
I20261116 08:58:39.994045 21628 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: Time spent bootstrapping tablet: real 0.045s user 0.025s sys 0.015s
I20261116 08:58:39.997145 21628 raft_consensus.cc:360] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "e6a10295c56d4db093049f2bf08fcea9" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45095 } }
I20261116 08:58:39.998026 21628 raft_consensus.cc:748] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a10295c56d4db093049f2bf08fcea9, State: Initialized, Role: FOLLOWER
I20261116 08:58:39.998848 21628 consensus_queue.cc:261] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e6a10295c56d4db093049f2bf08fcea9" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45095 } }
I20261116 08:58:39.999576 21628 raft_consensus.cc:400] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:39.999920 21628 raft_consensus.cc:499] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:40.000304 21628 raft_consensus.cc:3096] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:40.007809 21628 raft_consensus.cc:521] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "e6a10295c56d4db093049f2bf08fcea9" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45095 } }
I20261116 08:58:40.008407 21628 leader_election.cc:304] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6a10295c56d4db093049f2bf08fcea9; no voters:
I20261116 08:58:40.010303 21628 leader_election.cc:290] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20261116 08:58:40.010846 21659 raft_consensus.cc:2826] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 2 FOLLOWER]: Leader election won for term 2
I20261116 08:58:40.019232 21659 raft_consensus.cc:705] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 2 LEADER]: Becoming Leader. State: Replica: e6a10295c56d4db093049f2bf08fcea9, State: Running, Role: LEADER
I20261116 08:58:40.019658 21628 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9: Time spent starting tablet: real 0.025s user 0.027s sys 0.001s
I20261116 08:58:40.020390 21659 consensus_queue.cc:238] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e6a10295c56d4db093049f2bf08fcea9" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45095 } }
I20261116 08:58:40.028879 21662 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e6a10295c56d4db093049f2bf08fcea9" committed_config { opid_index: -1 peers { permanent_uuid: "e6a10295c56d4db093049f2bf08fcea9" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45095 } } }
I20261116 08:58:40.030941 21662 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20261116 08:58:40.030405 21661 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader e6a10295c56d4db093049f2bf08fcea9. Latest consensus state: current_term: 2 leader_uuid: "e6a10295c56d4db093049f2bf08fcea9" committed_config { opid_index: -1 peers { permanent_uuid: "e6a10295c56d4db093049f2bf08fcea9" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 45095 } } }
I20261116 08:58:40.031945 21661 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20261116 08:58:40.044193 21670 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261116 08:58:40.044878 21670 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20261116 08:58:40.066404 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:45095
I20261116 08:58:40.066718 21697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:45095 every 8 connection(s)
I20261116 08:58:40.072892 21698 heartbeater.cc:345] Connected to a master server at 127.18.102.62:33601
I20261116 08:58:40.073464 21698 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:40.074254 18840 tablet_server.cc:179] TabletServer@127.18.102.1:45095 shutting down...
I20261116 08:58:40.074589 21698 heartbeater.cc:508] Master 127.18.102.62:33601 requested a full tablet report, sending...
I20261116 08:58:40.078739 21455 ts_manager.cc:210] Re-registered known tserver with Master: e6a10295c56d4db093049f2bf08fcea9 (127.18.102.1:45095)
W20261116 08:58:40.079483 21455 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261116 08:58:40.081343 21455 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32850
I20261116 08:58:40.084046 21698 heartbeater.cc:500] Master 127.18.102.62:33601 was elected leader, sending a full tablet report...
W20261116 08:58:40.088001 21455 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261116 08:58:40.103456 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:40.104214 18840 tablet_replica.cc:333] stopping tablet replica
I20261116 08:58:40.104790 18840 raft_consensus.cc:2261] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 2 LEADER]: Raft consensus shutting down.
I20261116 08:58:40.105197 18840 raft_consensus.cc:2290] T 11111111111111111111111111111111 P e6a10295c56d4db093049f2bf08fcea9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:40.118196 18840 tablet_server.cc:196] TabletServer@127.18.102.1:45095 shutdown complete.
I20261116 08:58:40.130303 18840 master.cc:585] Master@127.18.102.62:33601 shutting down...
I20261116 08:58:40.157312 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:40.158066 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:40.158461 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7fd313ca9e494eb2a2321e222168f93a: stopping tablet replica
I20261116 08:58:40.178715 18840 master.cc:607] Master@127.18.102.62:33601 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (857 ms)
[ RUN ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20261116 08:58:40.220983 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:44657
I20261116 08:58:40.222494 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:40.228688 21704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:40.230222 21705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:40.232990 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:40.233412 21707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:40.234248 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:40.234418 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:40.234642 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819520234606 us; error 0 us; skew 500 ppm
I20261116 08:58:40.235252 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:40.237689 18840 webserver.cc:577] Webserver started at http://127.18.102.62:44073/ using document root <none> and password file <none>
I20261116 08:58:40.238237 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:40.238474 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:40.238848 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:40.240025 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "211dee44f8a9495491df88089d606411"
format_stamp: "Formatted at 2026-11-16 08:58:40 on dist-test-slave-qt0z"
I20261116 08:58:40.244544 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20261116 08:58:40.247785 21712 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:40.248670 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261116 08:58:40.248920 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "211dee44f8a9495491df88089d606411"
format_stamp: "Formatted at 2026-11-16 08:58:40 on dist-test-slave-qt0z"
I20261116 08:58:40.249159 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:40.264358 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:40.265338 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:40.314481 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:44657
I20261116 08:58:40.314580 21763 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:44657 every 8 connection(s)
I20261116 08:58:40.318955 21764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:40.328492 21764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411: Bootstrap starting.
I20261116 08:58:40.333076 21764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:40.337204 21764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411: No bootstrap required, opened a new log
I20261116 08:58:40.339279 21764 raft_consensus.cc:360] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211dee44f8a9495491df88089d606411" member_type: VOTER }
I20261116 08:58:40.339684 21764 raft_consensus.cc:386] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:40.339895 21764 raft_consensus.cc:748] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 211dee44f8a9495491df88089d606411, State: Initialized, Role: FOLLOWER
I20261116 08:58:40.340567 21764 consensus_queue.cc:261] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211dee44f8a9495491df88089d606411" member_type: VOTER }
I20261116 08:58:40.340992 21764 raft_consensus.cc:400] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:40.341195 21764 raft_consensus.cc:499] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:40.341452 21764 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:40.345798 21764 raft_consensus.cc:521] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211dee44f8a9495491df88089d606411" member_type: VOTER }
I20261116 08:58:40.346232 21764 leader_election.cc:304] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 211dee44f8a9495491df88089d606411; no voters:
I20261116 08:58:40.347476 21764 leader_election.cc:290] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:40.347805 21767 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:40.349401 21767 raft_consensus.cc:705] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [term 1 LEADER]: Becoming Leader. State: Replica: 211dee44f8a9495491df88089d606411, State: Running, Role: LEADER
I20261116 08:58:40.350057 21767 consensus_queue.cc:238] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211dee44f8a9495491df88089d606411" member_type: VOTER }
I20261116 08:58:40.350845 21764 sys_catalog.cc:565] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:40.353839 21768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "211dee44f8a9495491df88089d606411" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211dee44f8a9495491df88089d606411" member_type: VOTER } }
I20261116 08:58:40.354386 21768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:40.362298 21773 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:58:40.362190 21769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 211dee44f8a9495491df88089d606411. Latest consensus state: current_term: 1 leader_uuid: "211dee44f8a9495491df88089d606411" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211dee44f8a9495491df88089d606411" member_type: VOTER } }
I20261116 08:58:40.362766 21769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:40.369887 21773 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:58:40.372776 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:58:40.379966 21773 catalog_manager.cc:1398] Generated new cluster ID: 1ac531fd05b447bd9cb991f103525fcd
I20261116 08:58:40.380208 21773 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:58:40.398049 21773 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:58:40.399574 21773 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:58:40.413569 21773 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411: Generated new TSK 0
I20261116 08:58:40.414135 21773 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:58:40.438752 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:40.444608 21786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:40.446128 21787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:40.448974 21789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:40.452090 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:40.452935 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:40.453130 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:40.453269 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819520453256 us; error 0 us; skew 500 ppm
I20261116 08:58:40.453862 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:40.456414 18840 webserver.cc:577] Webserver started at http://127.18.102.1:43335/ using document root <none> and password file <none>
I20261116 08:58:40.456930 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:40.457116 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:40.457378 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:40.458391 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "606f5603c28a45c1bbcaac7ed6386f71"
format_stamp: "Formatted at 2026-11-16 08:58:40 on dist-test-slave-qt0z"
I20261116 08:58:40.462860 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20261116 08:58:40.466310 21794 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:40.467248 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261116 08:58:40.467553 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "606f5603c28a45c1bbcaac7ed6386f71"
format_stamp: "Formatted at 2026-11-16 08:58:40 on dist-test-slave-qt0z"
I20261116 08:58:40.467835 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:40.480147 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:40.481050 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:40.499505 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:40.499835 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20261116 08:58:40.500092 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:40.500253 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:40.580236 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:36099
I20261116 08:58:40.580374 21864 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:36099 every 8 connection(s)
I20261116 08:58:40.585014 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261116 08:58:40.585449 21865 heartbeater.cc:345] Connected to a master server at 127.18.102.62:44657
I20261116 08:58:40.585826 21865 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:40.586686 21865 heartbeater.cc:508] Master 127.18.102.62:44657 requested a full tablet report, sending...
I20261116 08:58:40.589151 21729 ts_manager.cc:210] Registered new tserver with Master: 606f5603c28a45c1bbcaac7ed6386f71 (127.18.102.1:36099)
W20261116 08:58:40.591598 21867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:40.592103 21729 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46842
W20261116 08:58:40.593835 21868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:40.596776 21870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:40.597268 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:40.598075 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:40.598253 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:40.598400 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819520598369 us; error 0 us; skew 500 ppm
I20261116 08:58:40.598883 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:40.601462 18840 webserver.cc:577] Webserver started at http://127.18.102.2:38479/ using document root <none> and password file <none>
I20261116 08:58:40.601913 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:40.602079 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:40.602329 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:40.603441 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-1-root/instance:
uuid: "3edd9cfb19e24b1d8bd941ae6bdaacd4"
format_stamp: "Formatted at 2026-11-16 08:58:40 on dist-test-slave-qt0z"
I20261116 08:58:40.607827 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20261116 08:58:40.611021 21875 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:40.611902 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261116 08:58:40.612196 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-1-root
uuid: "3edd9cfb19e24b1d8bd941ae6bdaacd4"
format_stamp: "Formatted at 2026-11-16 08:58:40 on dist-test-slave-qt0z"
I20261116 08:58:40.612468 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:40.635293 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:40.636252 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:40.643028 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:40.643379 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20261116 08:58:40.643709 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:40.643891 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:40.707825 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.2:46659
I20261116 08:58:40.707984 21944 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.2:46659 every 8 connection(s)
I20261116 08:58:40.713131 21945 heartbeater.cc:345] Connected to a master server at 127.18.102.62:44657
I20261116 08:58:40.713575 21945 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:40.714555 21945 heartbeater.cc:508] Master 127.18.102.62:44657 requested a full tablet report, sending...
I20261116 08:58:40.716697 21729 ts_manager.cc:210] Registered new tserver with Master: 3edd9cfb19e24b1d8bd941ae6bdaacd4 (127.18.102.2:46659)
I20261116 08:58:40.717347 18840 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.006142279s
I20261116 08:58:40.718640 21729 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46856
I20261116 08:58:40.753377 21830 tablet_service.cc:1500] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261116 08:58:40.754762 21830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:40.765694 21955 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: Bootstrap starting.
I20261116 08:58:40.772176 21955 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:40.777194 21955 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: No bootstrap required, opened a new log
I20261116 08:58:40.777619 21955 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20261116 08:58:40.780368 21955 raft_consensus.cc:360] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } }
I20261116 08:58:40.780886 21955 raft_consensus.cc:386] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:40.781149 21955 raft_consensus.cc:748] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606f5603c28a45c1bbcaac7ed6386f71, State: Initialized, Role: FOLLOWER
I20261116 08:58:40.781805 21955 consensus_queue.cc:261] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } }
I20261116 08:58:40.782449 21955 raft_consensus.cc:400] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:40.782733 21955 raft_consensus.cc:499] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:40.783038 21955 raft_consensus.cc:3096] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:40.789719 21955 raft_consensus.cc:521] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } }
I20261116 08:58:40.790346 21955 leader_election.cc:304] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 606f5603c28a45c1bbcaac7ed6386f71; no voters:
I20261116 08:58:40.791657 21955 leader_election.cc:290] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:40.792119 21957 raft_consensus.cc:2826] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:40.793663 21865 heartbeater.cc:500] Master 127.18.102.62:44657 was elected leader, sending a full tablet report...
I20261116 08:58:40.793962 21955 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: Time spent starting tablet: real 0.016s user 0.007s sys 0.010s
I20261116 08:58:40.795238 21957 raft_consensus.cc:705] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 1 LEADER]: Becoming Leader. State: Replica: 606f5603c28a45c1bbcaac7ed6386f71, State: Running, Role: LEADER
I20261116 08:58:40.795910 21957 consensus_queue.cc:238] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } }
I20261116 08:58:40.799810 21959 tablet_replica.cc:442] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: TxnStatusTablet state changed. Reason: New leader 606f5603c28a45c1bbcaac7ed6386f71. Latest consensus state: current_term: 1 leader_uuid: "606f5603c28a45c1bbcaac7ed6386f71" committed_config { opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } } }
I20261116 08:58:40.800320 21959 tablet_replica.cc:445] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: This TxnStatusTablet replica's current role is: LEADER
I20261116 08:58:40.803422 21958 tablet_replica.cc:442] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "606f5603c28a45c1bbcaac7ed6386f71" committed_config { opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } } }
I20261116 08:58:40.804095 21958 tablet_replica.cc:445] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: This TxnStatusTablet replica's current role is: LEADER
W20261116 08:58:40.806985 21729 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261116 08:58:40.814270 21962 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261116 08:58:40.814738 21962 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20261116 08:58:40.816144 21729 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261116 08:58:40.864051 21966 ts_tablet_manager.cc:979] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: Initiating tablet copy from peer 606f5603c28a45c1bbcaac7ed6386f71 (127.18.102.1:36099)
I20261116 08:58:40.865603 21966 tablet_copy_client.cc:323] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: tablet copy: Beginning tablet copy session from remote peer at address 127.18.102.1:36099
I20261116 08:58:40.874058 21850 tablet_copy_service.cc:140] P 606f5603c28a45c1bbcaac7ed6386f71: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer 3edd9cfb19e24b1d8bd941ae6bdaacd4 ({username='slave'} at 127.0.0.1:53616)
I20261116 08:58:40.874512 21850 tablet_copy_service.cc:161] P 606f5603c28a45c1bbcaac7ed6386f71: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer 3edd9cfb19e24b1d8bd941ae6bdaacd4 at {username='slave'} at 127.0.0.1:53616: session id = 3edd9cfb19e24b1d8bd941ae6bdaacd4-11111111111111111111111111111111
I20261116 08:58:40.879186 21850 tablet_copy_source_session.cc:215] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: Tablet Copy: opened 0 blocks and 1 log segments
I20261116 08:58:40.881524 21966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:40.890836 21966 tablet_copy_client.cc:806] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: tablet copy: Starting download of 0 data blocks...
I20261116 08:58:40.891360 21966 tablet_copy_client.cc:670] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: tablet copy: Starting download of 1 WAL segments...
I20261116 08:58:40.894721 21966 tablet_copy_client.cc:538] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261116 08:58:40.899915 21966 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: Bootstrap starting.
I20261116 08:58:40.923290 21966 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261116 08:58:40.923995 21966 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: Bootstrap complete.
I20261116 08:58:40.924521 21966 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.005s
I20261116 08:58:40.926483 21966 raft_consensus.cc:360] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } }
I20261116 08:58:40.926929 21966 raft_consensus.cc:748] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 3edd9cfb19e24b1d8bd941ae6bdaacd4, State: Initialized, Role: NON_PARTICIPANT
I20261116 08:58:40.927417 21966 consensus_queue.cc:261] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } }
I20261116 08:58:40.928925 21969 tablet_replica.cc:442] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } } }
I20261116 08:58:40.929085 21945 heartbeater.cc:500] Master 127.18.102.62:44657 was elected leader, sending a full tablet report...
I20261116 08:58:40.929322 21966 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20261116 08:58:40.929375 21969 tablet_replica.cc:445] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20261116 08:58:40.931444 21850 tablet_copy_service.cc:342] P 606f5603c28a45c1bbcaac7ed6386f71: Request end of tablet copy session 3edd9cfb19e24b1d8bd941ae6bdaacd4-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:53616
I20261116 08:58:40.931777 21850 tablet_copy_service.cc:434] P 606f5603c28a45c1bbcaac7ed6386f71: ending tablet copy session 3edd9cfb19e24b1d8bd941ae6bdaacd4-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer 3edd9cfb19e24b1d8bd941ae6bdaacd4
W20261116 08:58:40.932049 21729 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
W20261116 08:58:40.936195 21729 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261116 08:58:40.956005 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
I20261116 08:58:40.979535 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:40.980437 18840 tablet_replica.cc:333] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: stopping tablet replica
I20261116 08:58:40.981144 18840 raft_consensus.cc:2261] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:40.981773 18840 raft_consensus.cc:2290] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:41.004392 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
I20261116 08:58:41.016014 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:41.022027 21970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:41.023031 21971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:41.025604 21973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:41.026013 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:41.026988 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:41.027235 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:41.027416 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819521027381 us; error 0 us; skew 500 ppm
I20261116 08:58:41.027952 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:41.035307 18840 webserver.cc:577] Webserver started at http://127.18.102.1:43335/ using document root <none> and password file <none>
I20261116 08:58:41.035835 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:41.036015 18840 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261116 08:58:41.039794 18840 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20261116 08:58:41.042646 21978 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:41.043541 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261116 08:58:41.043880 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "606f5603c28a45c1bbcaac7ed6386f71"
format_stamp: "Formatted at 2026-11-16 08:58:40 on dist-test-slave-qt0z"
I20261116 08:58:41.044241 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:41.074623 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:41.076004 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:41.084105 21987 ts_tablet_manager.cc:594] Loading tablet metadata (0/1 complete)
I20261116 08:58:41.091041 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261116 08:58:41.091372 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.001s
I20261116 08:58:41.091665 18840 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261116 08:58:41.097518 18840 ts_tablet_manager.cc:662] Registered 1 tablets
I20261116 08:58:41.097750 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.006s user 0.004s sys 0.000s
I20261116 08:58:41.097971 21987 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: Bootstrap starting.
I20261116 08:58:41.147735 21987 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261116 08:58:41.149657 21987 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: Bootstrap complete.
I20261116 08:58:41.150646 21987 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: Time spent bootstrapping tablet: real 0.053s user 0.042s sys 0.008s
I20261116 08:58:41.154747 21987 raft_consensus.cc:360] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } }
I20261116 08:58:41.155706 21987 raft_consensus.cc:748] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606f5603c28a45c1bbcaac7ed6386f71, State: Initialized, Role: FOLLOWER
I20261116 08:58:41.156895 21987 consensus_queue.cc:261] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } }
I20261116 08:58:41.157920 21987 raft_consensus.cc:400] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:41.158344 21987 raft_consensus.cc:499] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:41.158814 21987 raft_consensus.cc:3096] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 1 FOLLOWER]: Advancing to term 2
I20261116 08:58:41.167006 21987 raft_consensus.cc:521] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } }
I20261116 08:58:41.167685 21987 leader_election.cc:304] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 606f5603c28a45c1bbcaac7ed6386f71; no voters:
I20261116 08:58:41.176421 21987 leader_election.cc:290] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [CANDIDATE]: Term 2 election: Requested vote from peers
I20261116 08:58:41.177510 22009 raft_consensus.cc:2826] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 2 FOLLOWER]: Leader election won for term 2
I20261116 08:58:41.194571 21987 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71: Time spent starting tablet: real 0.043s user 0.026s sys 0.013s
I20261116 08:58:41.198864 22009 raft_consensus.cc:705] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 2 LEADER]: Becoming Leader. State: Replica: 606f5603c28a45c1bbcaac7ed6386f71, State: Running, Role: LEADER
I20261116 08:58:41.200377 22009 consensus_queue.cc:238] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } }
I20261116 08:58:41.212558 22016 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "606f5603c28a45c1bbcaac7ed6386f71" committed_config { opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } } }
I20261116 08:58:41.213228 22016 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20261116 08:58:41.214775 22014 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 606f5603c28a45c1bbcaac7ed6386f71. Latest consensus state: current_term: 2 leader_uuid: "606f5603c28a45c1bbcaac7ed6386f71" committed_config { opid_index: -1 peers { permanent_uuid: "606f5603c28a45c1bbcaac7ed6386f71" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 36099 } } }
I20261116 08:58:41.215413 22014 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20261116 08:58:41.225844 22021 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261116 08:58:41.226354 22021 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20261116 08:58:41.257572 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:36099
I20261116 08:58:41.257630 22056 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:36099 every 8 connection(s)
I20261116 08:58:41.265170 22057 heartbeater.cc:345] Connected to a master server at 127.18.102.62:44657
I20261116 08:58:41.265357 18840 tablet_server.cc:179] TabletServer@127.18.102.1:36099 shutting down...
I20261116 08:58:41.265585 22057 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:41.268549 22057 heartbeater.cc:508] Master 127.18.102.62:44657 requested a full tablet report, sending...
I20261116 08:58:41.273588 21729 ts_manager.cc:210] Re-registered known tserver with Master: 606f5603c28a45c1bbcaac7ed6386f71 (127.18.102.1:36099)
W20261116 08:58:41.274137 21729 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261116 08:58:41.276659 21729 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46880
I20261116 08:58:41.285571 22057 heartbeater.cc:500] Master 127.18.102.62:44657 was elected leader, sending a full tablet report...
W20261116 08:58:41.290958 21729 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261116 08:58:41.292938 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:41.293768 18840 tablet_replica.cc:333] stopping tablet replica
I20261116 08:58:41.294315 18840 raft_consensus.cc:2261] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 2 LEADER]: Raft consensus shutting down.
I20261116 08:58:41.294750 18840 raft_consensus.cc:2290] T 11111111111111111111111111111111 P 606f5603c28a45c1bbcaac7ed6386f71 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:41.305130 18840 tablet_server.cc:196] TabletServer@127.18.102.1:36099 shutdown complete.
I20261116 08:58:41.314937 18840 tablet_server.cc:179] TabletServer@127.18.102.2:0 shutting down...
I20261116 08:58:41.335932 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:41.336687 18840 tablet_replica.cc:333] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4: stopping tablet replica
I20261116 08:58:41.337244 18840 raft_consensus.cc:2261] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20261116 08:58:41.337642 18840 raft_consensus.cc:2290] T 11111111111111111111111111111111 P 3edd9cfb19e24b1d8bd941ae6bdaacd4 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20261116 08:58:41.356491 18840 tablet_server.cc:196] TabletServer@127.18.102.2:0 shutdown complete.
I20261116 08:58:41.366828 18840 master.cc:585] Master@127.18.102.62:44657 shutting down...
I20261116 08:58:41.381894 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:41.382359 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:41.382651 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P 211dee44f8a9495491df88089d606411: stopping tablet replica
I20261116 08:58:41.400604 18840 master.cc:607] Master@127.18.102.62:44657 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1205 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20261116 08:58:41.425170 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:35227
I20261116 08:58:41.426219 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:41.431567 22063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:41.432004 22064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:41.434719 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:41.434976 22066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:41.435822 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:41.435988 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:41.436160 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819521436148 us; error 0 us; skew 500 ppm
I20261116 08:58:41.436702 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:41.439332 18840 webserver.cc:577] Webserver started at http://127.18.102.62:46231/ using document root <none> and password file <none>
I20261116 08:58:41.439802 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:41.439968 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:41.440232 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:41.441229 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "690d54f18efc484c8413f3fdfd799b7b"
format_stamp: "Formatted at 2026-11-16 08:58:41 on dist-test-slave-qt0z"
I20261116 08:58:41.445854 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20261116 08:58:41.449152 22071 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:41.449924 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261116 08:58:41.450188 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "690d54f18efc484c8413f3fdfd799b7b"
format_stamp: "Formatted at 2026-11-16 08:58:41 on dist-test-slave-qt0z"
I20261116 08:58:41.450420 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:41.474969 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:41.476017 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:41.525951 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:35227
I20261116 08:58:41.526034 22122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:35227 every 8 connection(s)
I20261116 08:58:41.530347 22123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:41.544917 22123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b: Bootstrap starting.
I20261116 08:58:41.551097 22123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:41.555919 22123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b: No bootstrap required, opened a new log
I20261116 08:58:41.558499 22123 raft_consensus.cc:360] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690d54f18efc484c8413f3fdfd799b7b" member_type: VOTER }
I20261116 08:58:41.559015 22123 raft_consensus.cc:386] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:41.559330 22123 raft_consensus.cc:748] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 690d54f18efc484c8413f3fdfd799b7b, State: Initialized, Role: FOLLOWER
I20261116 08:58:41.560003 22123 consensus_queue.cc:261] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690d54f18efc484c8413f3fdfd799b7b" member_type: VOTER }
I20261116 08:58:41.560577 22123 raft_consensus.cc:400] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:41.560865 22123 raft_consensus.cc:499] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:41.561177 22123 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:41.567569 22123 raft_consensus.cc:521] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690d54f18efc484c8413f3fdfd799b7b" member_type: VOTER }
I20261116 08:58:41.568183 22123 leader_election.cc:304] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 690d54f18efc484c8413f3fdfd799b7b; no voters:
I20261116 08:58:41.569499 22123 leader_election.cc:290] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:41.569748 22126 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:41.571713 22126 raft_consensus.cc:705] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [term 1 LEADER]: Becoming Leader. State: Replica: 690d54f18efc484c8413f3fdfd799b7b, State: Running, Role: LEADER
I20261116 08:58:41.572395 22126 consensus_queue.cc:238] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690d54f18efc484c8413f3fdfd799b7b" member_type: VOTER }
I20261116 08:58:41.572775 22123 sys_catalog.cc:565] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:41.574954 22128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 690d54f18efc484c8413f3fdfd799b7b. Latest consensus state: current_term: 1 leader_uuid: "690d54f18efc484c8413f3fdfd799b7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690d54f18efc484c8413f3fdfd799b7b" member_type: VOTER } }
I20261116 08:58:41.574941 22127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "690d54f18efc484c8413f3fdfd799b7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690d54f18efc484c8413f3fdfd799b7b" member_type: VOTER } }
I20261116 08:58:41.575589 22128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:41.575717 22127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:41.585060 22131 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:58:41.590133 22131 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:58:41.594875 22142 catalog_manager.cc:820] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20261116 08:58:41.595335 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:58:41.600298 22131 catalog_manager.cc:1398] Generated new cluster ID: ca4d5c7329d74dcd921ccf9c2fba9146
I20261116 08:58:41.600522 22131 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:58:41.612670 22131 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:58:41.613760 22131 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:58:41.626020 22131 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b: Generated new TSK 0
I20261116 08:58:41.626506 22131 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:58:41.661298 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:41.667371 22144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:41.668712 22145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:41.671212 22147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:41.671396 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:41.672286 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:41.672501 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:41.672680 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819521672663 us; error 0 us; skew 500 ppm
I20261116 08:58:41.673209 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:41.676066 18840 webserver.cc:577] Webserver started at http://127.18.102.1:41583/ using document root <none> and password file <none>
I20261116 08:58:41.676522 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:41.676738 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:41.677002 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:41.678035 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "d10e57c519174591a37945115cb20654"
format_stamp: "Formatted at 2026-11-16 08:58:41 on dist-test-slave-qt0z"
I20261116 08:58:41.682605 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20261116 08:58:41.685976 22152 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:41.686838 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261116 08:58:41.687109 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "d10e57c519174591a37945115cb20654"
format_stamp: "Formatted at 2026-11-16 08:58:41 on dist-test-slave-qt0z"
I20261116 08:58:41.687408 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:41.715669 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:41.716648 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:41.724938 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:41.725302 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:41.725610 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:41.725823 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:41.783210 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:35185
I20261116 08:58:41.783321 22222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:35185 every 8 connection(s)
I20261116 08:58:41.788869 22223 heartbeater.cc:345] Connected to a master server at 127.18.102.62:35227
I20261116 08:58:41.789283 22223 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:41.789958 22223 heartbeater.cc:508] Master 127.18.102.62:35227 requested a full tablet report, sending...
I20261116 08:58:41.791887 22088 ts_manager.cc:210] Registered new tserver with Master: d10e57c519174591a37945115cb20654 (127.18.102.1:35185)
I20261116 08:58:41.792152 18840 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004972366s
I20261116 08:58:41.793586 22088 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46724
I20261116 08:58:41.828970 22188 tablet_service.cc:1500] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261116 08:58:41.830397 22188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:41.845638 22233 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654: Bootstrap starting.
I20261116 08:58:41.850044 22233 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:41.854255 22233 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654: No bootstrap required, opened a new log
I20261116 08:58:41.854651 22233 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20261116 08:58:41.856527 22233 raft_consensus.cc:360] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "d10e57c519174591a37945115cb20654" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35185 } }
I20261116 08:58:41.856927 22233 raft_consensus.cc:386] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:41.857142 22233 raft_consensus.cc:748] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e57c519174591a37945115cb20654, State: Initialized, Role: FOLLOWER
I20261116 08:58:41.857702 22233 consensus_queue.cc:261] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "d10e57c519174591a37945115cb20654" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35185 } }
I20261116 08:58:41.858157 22233 raft_consensus.cc:400] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:41.858405 22233 raft_consensus.cc:499] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:41.858662 22233 raft_consensus.cc:3096] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:41.862998 22233 raft_consensus.cc:521] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "d10e57c519174591a37945115cb20654" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35185 } }
I20261116 08:58:41.863582 22233 leader_election.cc:304] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d10e57c519174591a37945115cb20654; no voters:
I20261116 08:58:41.864905 22233 leader_election.cc:290] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:41.865235 22235 raft_consensus.cc:2826] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:41.866822 22223 heartbeater.cc:500] Master 127.18.102.62:35227 was elected leader, sending a full tablet report...
I20261116 08:58:41.867012 22235 raft_consensus.cc:705] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [term 1 LEADER]: Becoming Leader. State: Replica: d10e57c519174591a37945115cb20654, State: Running, Role: LEADER
I20261116 08:58:41.867784 22235 consensus_queue.cc:238] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "d10e57c519174591a37945115cb20654" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35185 } }
I20261116 08:58:41.868476 22233 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20261116 08:58:41.871075 22237 tablet_replica.cc:442] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654: TxnStatusTablet state changed. Reason: New leader d10e57c519174591a37945115cb20654. Latest consensus state: current_term: 1 leader_uuid: "d10e57c519174591a37945115cb20654" committed_config { opid_index: -1 peers { permanent_uuid: "d10e57c519174591a37945115cb20654" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35185 } } }
I20261116 08:58:41.871358 22236 tablet_replica.cc:442] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d10e57c519174591a37945115cb20654" committed_config { opid_index: -1 peers { permanent_uuid: "d10e57c519174591a37945115cb20654" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35185 } } }
I20261116 08:58:41.871920 22236 tablet_replica.cc:445] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654: This TxnStatusTablet replica's current role is: LEADER
W20261116 08:58:41.875677 22088 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261116 08:58:41.871649 22237 tablet_replica.cc:445] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654: This TxnStatusTablet replica's current role is: LEADER
I20261116 08:58:41.880594 22239 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261116 08:58:41.881045 22239 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20261116 08:58:41.883522 22088 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
W20261116 08:58:41.932624 22170 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20261116 08:58:41.932942 22170 txn_status_manager.cc:244] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20261116 08:58:41.933271 22170 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20261116 08:58:41.941318 22170 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20261116 08:58:41.982010 22170 txn_status_manager.cc:244] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20261116 08:58:41.982405 22170 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20261116 08:58:41.990078 22170 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20261116 08:58:42.001796 22188 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:37542
W20261116 08:58:42.004142 22188 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:37542
W20261116 08:58:42.006099 22188 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:37542
W20261116 08:58:42.008183 22188 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:37542
W20261116 08:58:42.010140 22188 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:37542
W20261116 08:58:42.012226 22188 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:37542
W20261116 08:58:42.021971 22188 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:37550
W20261116 08:58:42.024185 22188 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:37550
W20261116 08:58:42.026206 22188 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:37550
W20261116 08:58:42.028393 22188 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:37550
W20261116 08:58:42.030318 22188 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:37550
W20261116 08:58:42.032330 22188 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:37550
I20261116 08:58:42.034655 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
I20261116 08:58:42.055651 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:42.056456 18840 tablet_replica.cc:333] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654: stopping tablet replica
I20261116 08:58:42.057108 18840 raft_consensus.cc:2261] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:42.057638 18840 raft_consensus.cc:2290] T 11111111111111111111111111111111 P d10e57c519174591a37945115cb20654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:42.069695 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
I20261116 08:58:42.079821 18840 master.cc:585] Master@127.18.102.62:35227 shutting down...
I20261116 08:58:42.095395 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:42.095846 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:42.096292 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P 690d54f18efc484c8413f3fdfd799b7b: stopping tablet replica
I20261116 08:58:42.104657 18840 master.cc:607] Master@127.18.102.62:35227 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (702 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20261116 08:58:42.129278 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:44681
I20261116 08:58:42.130589 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:42.146042 22248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:42.149169 22249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:42.150213 22251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:42.151204 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:42.151952 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:42.152249 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:42.152429 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819522152411 us; error 0 us; skew 500 ppm
I20261116 08:58:42.153049 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:42.155822 18840 webserver.cc:577] Webserver started at http://127.18.102.62:43071/ using document root <none> and password file <none>
I20261116 08:58:42.156276 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:42.156450 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:42.156709 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:42.157716 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "86debd0e2dab460b9389913ac865bcae"
format_stamp: "Formatted at 2026-11-16 08:58:42 on dist-test-slave-qt0z"
I20261116 08:58:42.162091 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20261116 08:58:42.165482 22256 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:42.166283 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20261116 08:58:42.166550 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "86debd0e2dab460b9389913ac865bcae"
format_stamp: "Formatted at 2026-11-16 08:58:42 on dist-test-slave-qt0z"
I20261116 08:58:42.166805 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:42.180475 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:42.181354 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:42.213747 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:44681
I20261116 08:58:42.213829 22307 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:44681 every 8 connection(s)
I20261116 08:58:42.217311 22308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:42.226697 22308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae: Bootstrap starting.
I20261116 08:58:42.230609 22308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:42.234097 22308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae: No bootstrap required, opened a new log
I20261116 08:58:42.236001 22308 raft_consensus.cc:360] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86debd0e2dab460b9389913ac865bcae" member_type: VOTER }
I20261116 08:58:42.236333 22308 raft_consensus.cc:386] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:42.236501 22308 raft_consensus.cc:748] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86debd0e2dab460b9389913ac865bcae, State: Initialized, Role: FOLLOWER
I20261116 08:58:42.236958 22308 consensus_queue.cc:261] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86debd0e2dab460b9389913ac865bcae" member_type: VOTER }
I20261116 08:58:42.237332 22308 raft_consensus.cc:400] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:42.237509 22308 raft_consensus.cc:499] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:42.237705 22308 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:42.241315 22308 raft_consensus.cc:521] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86debd0e2dab460b9389913ac865bcae" member_type: VOTER }
I20261116 08:58:42.241736 22308 leader_election.cc:304] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86debd0e2dab460b9389913ac865bcae; no voters:
I20261116 08:58:42.242676 22308 leader_election.cc:290] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:42.242949 22311 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:42.244309 22311 raft_consensus.cc:705] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [term 1 LEADER]: Becoming Leader. State: Replica: 86debd0e2dab460b9389913ac865bcae, State: Running, Role: LEADER
I20261116 08:58:42.244930 22311 consensus_queue.cc:238] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86debd0e2dab460b9389913ac865bcae" member_type: VOTER }
I20261116 08:58:42.245472 22308 sys_catalog.cc:565] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:42.248202 22313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86debd0e2dab460b9389913ac865bcae. Latest consensus state: current_term: 1 leader_uuid: "86debd0e2dab460b9389913ac865bcae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86debd0e2dab460b9389913ac865bcae" member_type: VOTER } }
I20261116 08:58:42.248148 22312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86debd0e2dab460b9389913ac865bcae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86debd0e2dab460b9389913ac865bcae" member_type: VOTER } }
I20261116 08:58:42.248919 22313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:42.249190 22312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:42.254619 22317 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:58:42.260258 22317 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:58:42.261332 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:58:42.268225 22317 catalog_manager.cc:1398] Generated new cluster ID: 1912937b3d7c485f929730eda1570730
I20261116 08:58:42.268461 22317 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:58:42.292438 22317 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:58:42.293534 22317 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:58:42.306135 22317 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae: Generated new TSK 0
I20261116 08:58:42.306578 22317 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:58:42.327533 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:42.332893 22329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:42.333817 22330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:42.335839 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:42.336246 22332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:42.337021 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:42.337231 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:42.337399 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819522337366 us; error 0 us; skew 500 ppm
I20261116 08:58:42.337894 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:42.340068 18840 webserver.cc:577] Webserver started at http://127.18.102.1:44629/ using document root <none> and password file <none>
I20261116 08:58:42.340545 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:42.340749 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:42.341006 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:42.342018 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "87fd73a2a8534f6a83a7a05bff2a779f"
format_stamp: "Formatted at 2026-11-16 08:58:42 on dist-test-slave-qt0z"
I20261116 08:58:42.346217 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20261116 08:58:42.349298 22337 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:42.350063 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261116 08:58:42.350445 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "87fd73a2a8534f6a83a7a05bff2a779f"
format_stamp: "Formatted at 2026-11-16 08:58:42 on dist-test-slave-qt0z"
I20261116 08:58:42.350760 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:42.361965 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:42.362870 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:42.367623 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:42.367962 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:42.368392 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:42.368619 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:42.414603 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:34863
I20261116 08:58:42.414700 22407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:34863 every 8 connection(s)
I20261116 08:58:42.419211 22408 heartbeater.cc:345] Connected to a master server at 127.18.102.62:44681
I20261116 08:58:42.419533 22408 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:42.420167 22408 heartbeater.cc:508] Master 127.18.102.62:44681 requested a full tablet report, sending...
I20261116 08:58:42.422011 22273 ts_manager.cc:210] Registered new tserver with Master: 87fd73a2a8534f6a83a7a05bff2a779f (127.18.102.1:34863)
I20261116 08:58:42.422574 18840 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004813717s
I20261116 08:58:42.423687 22273 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51130
I20261116 08:58:42.451130 22373 tablet_service.cc:1500] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261116 08:58:42.452257 22373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:42.462680 22418 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f: Bootstrap starting.
I20261116 08:58:42.466909 22418 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:42.470561 22418 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f: No bootstrap required, opened a new log
I20261116 08:58:42.470934 22418 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20261116 08:58:42.472810 22418 raft_consensus.cc:360] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "87fd73a2a8534f6a83a7a05bff2a779f" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 34863 } }
I20261116 08:58:42.473233 22418 raft_consensus.cc:386] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:42.473470 22418 raft_consensus.cc:748] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87fd73a2a8534f6a83a7a05bff2a779f, State: Initialized, Role: FOLLOWER
I20261116 08:58:42.474047 22418 consensus_queue.cc:261] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "87fd73a2a8534f6a83a7a05bff2a779f" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 34863 } }
I20261116 08:58:42.474514 22418 raft_consensus.cc:400] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:42.474761 22418 raft_consensus.cc:499] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:42.475013 22418 raft_consensus.cc:3096] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:42.479697 22418 raft_consensus.cc:521] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "87fd73a2a8534f6a83a7a05bff2a779f" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 34863 } }
I20261116 08:58:42.480208 22418 leader_election.cc:304] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87fd73a2a8534f6a83a7a05bff2a779f; no voters:
I20261116 08:58:42.481232 22418 leader_election.cc:290] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:42.481529 22420 raft_consensus.cc:2826] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:42.482828 22420 raft_consensus.cc:705] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [term 1 LEADER]: Becoming Leader. State: Replica: 87fd73a2a8534f6a83a7a05bff2a779f, State: Running, Role: LEADER
I20261116 08:58:42.482997 22408 heartbeater.cc:500] Master 127.18.102.62:44681 was elected leader, sending a full tablet report...
I20261116 08:58:42.483587 22418 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20261116 08:58:42.483590 22420 consensus_queue.cc:238] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "87fd73a2a8534f6a83a7a05bff2a779f" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 34863 } }
I20261116 08:58:42.489298 22422 tablet_replica.cc:442] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f: TxnStatusTablet state changed. Reason: New leader 87fd73a2a8534f6a83a7a05bff2a779f. Latest consensus state: current_term: 1 leader_uuid: "87fd73a2a8534f6a83a7a05bff2a779f" committed_config { opid_index: -1 peers { permanent_uuid: "87fd73a2a8534f6a83a7a05bff2a779f" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 34863 } } }
I20261116 08:58:42.489352 22421 tablet_replica.cc:442] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87fd73a2a8534f6a83a7a05bff2a779f" committed_config { opid_index: -1 peers { permanent_uuid: "87fd73a2a8534f6a83a7a05bff2a779f" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 34863 } } }
I20261116 08:58:42.489796 22422 tablet_replica.cc:445] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f: This TxnStatusTablet replica's current role is: LEADER
I20261116 08:58:42.489951 22421 tablet_replica.cc:445] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f: This TxnStatusTablet replica's current role is: LEADER
W20261116 08:58:42.491235 22273 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261116 08:58:42.491297 22425 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261116 08:58:42.491874 22425 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20261116 08:58:42.495469 22273 catalog_manager.cc:5673] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261116 08:58:42.519637 22373 tablet_service.cc:1500] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261116 08:58:42.521296 22373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:42.530912 22418 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f: Bootstrap starting.
I20261116 08:58:42.534984 22418 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:42.538412 22418 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f: No bootstrap required, opened a new log
I20261116 08:58:42.538717 22418 ts_tablet_manager.cc:1449] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20261116 08:58:42.540501 22418 raft_consensus.cc:360] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "87fd73a2a8534f6a83a7a05bff2a779f" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 34863 } }
I20261116 08:58:42.540834 22418 raft_consensus.cc:386] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:42.540989 22418 raft_consensus.cc:748] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87fd73a2a8534f6a83a7a05bff2a779f, State: Initialized, Role: FOLLOWER
I20261116 08:58:42.541378 22418 consensus_queue.cc:261] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "87fd73a2a8534f6a83a7a05bff2a779f" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 34863 } }
I20261116 08:58:42.541723 22418 raft_consensus.cc:400] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:42.541886 22418 raft_consensus.cc:499] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:42.542062 22418 raft_consensus.cc:3096] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:42.545936 22418 raft_consensus.cc:521] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "87fd73a2a8534f6a83a7a05bff2a779f" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 34863 } }
I20261116 08:58:42.546371 22418 leader_election.cc:304] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87fd73a2a8534f6a83a7a05bff2a779f; no voters:
I20261116 08:58:42.546878 22418 leader_election.cc:290] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:42.547009 22422 raft_consensus.cc:2826] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:42.547602 22422 raft_consensus.cc:705] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [term 1 LEADER]: Becoming Leader. State: Replica: 87fd73a2a8534f6a83a7a05bff2a779f, State: Running, Role: LEADER
I20261116 08:58:42.548323 22418 ts_tablet_manager.cc:1480] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20261116 08:58:42.548199 22422 consensus_queue.cc:238] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "87fd73a2a8534f6a83a7a05bff2a779f" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 34863 } }
W20261116 08:58:42.552773 22273 catalog_manager.cc:5673] Ignoring report from unknown tablet 22222222222222222222222222222222
W20261116 08:58:42.556826 22273 catalog_manager.cc:5673] Ignoring report from unknown tablet 22222222222222222222222222222222
I20261116 08:58:42.579468 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
I20261116 08:58:42.605093 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:42.605890 18840 tablet_replica.cc:333] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f: stopping tablet replica
I20261116 08:58:42.606537 18840 raft_consensus.cc:2261] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:42.606957 18840 raft_consensus.cc:2290] T 22222222222222222222222222222222 P 87fd73a2a8534f6a83a7a05bff2a779f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:42.609381 18840 tablet_replica.cc:333] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f: stopping tablet replica
I20261116 08:58:42.609841 18840 raft_consensus.cc:2261] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:42.610249 18840 raft_consensus.cc:2290] T 11111111111111111111111111111111 P 87fd73a2a8534f6a83a7a05bff2a779f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:42.629954 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
I20261116 08:58:42.640944 18840 master.cc:585] Master@127.18.102.62:44681 shutting down...
I20261116 08:58:42.653942 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:42.654528 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:42.654951 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P 86debd0e2dab460b9389913ac865bcae: stopping tablet replica
I20261116 08:58:42.663578 18840 master.cc:607] Master@127.18.102.62:44681 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (558 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (3325 ms total)
[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/0
I20261116 08:58:42.685629 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:42867
I20261116 08:58:42.686511 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:42.690732 22434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:42.691874 22435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:42.693684 22437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:42.693951 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:42.694638 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:42.694891 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:42.695053 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819522695038 us; error 0 us; skew 500 ppm
I20261116 08:58:42.695621 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:42.697679 18840 webserver.cc:577] Webserver started at http://127.18.102.62:39239/ using document root <none> and password file <none>
I20261116 08:58:42.698153 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:42.698343 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:42.698603 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:42.699643 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "51add35177144b1181d225e6cd8c3a9e"
format_stamp: "Formatted at 2026-11-16 08:58:42 on dist-test-slave-qt0z"
I20261116 08:58:42.703533 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20261116 08:58:42.706458 22442 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:42.707227 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261116 08:58:42.707501 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "51add35177144b1181d225e6cd8c3a9e"
format_stamp: "Formatted at 2026-11-16 08:58:42 on dist-test-slave-qt0z"
I20261116 08:58:42.707782 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:42.717676 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:42.718552 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:42.751259 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:42867
I20261116 08:58:42.751374 22493 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:42867 every 8 connection(s)
I20261116 08:58:42.755285 22494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:42.764534 22494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e: Bootstrap starting.
I20261116 08:58:42.768775 22494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:42.772213 22494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e: No bootstrap required, opened a new log
I20261116 08:58:42.774070 22494 raft_consensus.cc:360] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51add35177144b1181d225e6cd8c3a9e" member_type: VOTER }
I20261116 08:58:42.774441 22494 raft_consensus.cc:386] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:42.774662 22494 raft_consensus.cc:748] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51add35177144b1181d225e6cd8c3a9e, State: Initialized, Role: FOLLOWER
I20261116 08:58:42.775276 22494 consensus_queue.cc:261] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51add35177144b1181d225e6cd8c3a9e" member_type: VOTER }
I20261116 08:58:42.775725 22494 raft_consensus.cc:400] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:42.775929 22494 raft_consensus.cc:499] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:42.776180 22494 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:42.780066 22494 raft_consensus.cc:521] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51add35177144b1181d225e6cd8c3a9e" member_type: VOTER }
I20261116 08:58:42.780544 22494 leader_election.cc:304] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51add35177144b1181d225e6cd8c3a9e; no voters:
I20261116 08:58:42.781565 22494 leader_election.cc:290] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:42.781859 22497 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:42.783044 22497 raft_consensus.cc:705] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [term 1 LEADER]: Becoming Leader. State: Replica: 51add35177144b1181d225e6cd8c3a9e, State: Running, Role: LEADER
I20261116 08:58:42.783798 22497 consensus_queue.cc:238] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51add35177144b1181d225e6cd8c3a9e" member_type: VOTER }
I20261116 08:58:42.784574 22494 sys_catalog.cc:565] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:42.785924 22498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51add35177144b1181d225e6cd8c3a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51add35177144b1181d225e6cd8c3a9e" member_type: VOTER } }
I20261116 08:58:42.786360 22498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:42.787552 22499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51add35177144b1181d225e6cd8c3a9e. Latest consensus state: current_term: 1 leader_uuid: "51add35177144b1181d225e6cd8c3a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51add35177144b1181d225e6cd8c3a9e" member_type: VOTER } }
I20261116 08:58:42.788075 22499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:42.789976 22501 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:58:42.794407 22501 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:58:42.799115 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:58:42.803236 22501 catalog_manager.cc:1398] Generated new cluster ID: 26e2629381f948d2b92eebd62c0ae06d
I20261116 08:58:42.803515 22501 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:58:42.818451 22501 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:58:42.819521 22501 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:58:42.831703 22501 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e: Generated new TSK 0
I20261116 08:58:42.832269 22501 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:58:42.865162 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:42.871222 22515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:42.872313 22516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:42.873760 22518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:42.874171 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:42.874912 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:42.875094 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:42.875320 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819522875291 us; error 0 us; skew 500 ppm
I20261116 08:58:42.875808 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:42.877843 18840 webserver.cc:577] Webserver started at http://127.18.102.1:38717/ using document root <none> and password file <none>
I20261116 08:58:42.878296 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:42.878486 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:42.878742 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:42.879805 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "43e7664954cb4fa087e0575a1d1c9fec"
format_stamp: "Formatted at 2026-11-16 08:58:42 on dist-test-slave-qt0z"
I20261116 08:58:42.883973 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20261116 08:58:42.886986 22523 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:42.887698 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261116 08:58:42.887991 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "43e7664954cb4fa087e0575a1d1c9fec"
format_stamp: "Formatted at 2026-11-16 08:58:42 on dist-test-slave-qt0z"
I20261116 08:58:42.888267 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:42.902624 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:42.903604 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:42.904899 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:42.906872 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:42.907073 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:42.907330 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:42.907492 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:42.942840 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:42633
I20261116 08:58:42.942917 22585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:42633 every 8 connection(s)
I20261116 08:58:42.947677 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:42.953711 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:42.954911 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:42.960707 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:42.960847 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:42.961601 22586 heartbeater.cc:345] Connected to a master server at 127.18.102.62:42867
I20261116 08:58:42.961858 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20261116 08:58:42.961915 22586 heartbeater.cc:462] Registering TS with master...
W20261116 08:58:42.962198 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:42.962395 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819522962363 us; error 0 us; skew 500 ppm
I20261116 08:58:42.962769 22586 heartbeater.cc:508] Master 127.18.102.62:42867 requested a full tablet report, sending...
I20261116 08:58:42.962951 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:42.964845 22459 ts_manager.cc:210] Registered new tserver with Master: 43e7664954cb4fa087e0575a1d1c9fec (127.18.102.1:42633)
I20261116 08:58:42.966334 22459 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57754
I20261116 08:58:42.966953 18840 webserver.cc:577] Webserver started at http://127.18.102.2:44073/ using document root <none> and password file <none>
I20261116 08:58:42.967569 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:42.967787 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:42.968068 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:42.969158 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-1-root/instance:
uuid: "bd4f754a75d249769ec22f66b514001c"
format_stamp: "Formatted at 2026-11-16 08:58:42 on dist-test-slave-qt0z"
I20261116 08:58:42.973609 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20261116 08:58:42.976516 22598 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:42.977212 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261116 08:58:42.977483 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-1-root
uuid: "bd4f754a75d249769ec22f66b514001c"
format_stamp: "Formatted at 2026-11-16 08:58:42 on dist-test-slave-qt0z"
I20261116 08:58:42.977748 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:42.998492 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:42.999490 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:43.000789 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:43.002808 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:43.003017 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:43.003266 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:43.003456 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:43.038667 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.2:43391
I20261116 08:58:43.038733 22660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.2:43391 every 8 connection(s)
I20261116 08:58:43.043345 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:43.049126 22664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:43.049623 22665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:43.051517 22667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:43.052970 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:43.053998 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:43.054255 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:43.054469 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819523054444 us; error 0 us; skew 500 ppm
I20261116 08:58:43.054775 22661 heartbeater.cc:345] Connected to a master server at 127.18.102.62:42867
I20261116 08:58:43.055244 22661 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:43.055280 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:43.055958 22661 heartbeater.cc:508] Master 127.18.102.62:42867 requested a full tablet report, sending...
I20261116 08:58:43.057835 18840 webserver.cc:577] Webserver started at http://127.18.102.3:36041/ using document root <none> and password file <none>
I20261116 08:58:43.058081 22459 ts_manager.cc:210] Registered new tserver with Master: bd4f754a75d249769ec22f66b514001c (127.18.102.2:43391)
I20261116 08:58:43.058447 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:43.058715 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:43.059023 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:43.059839 22459 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57764
I20261116 08:58:43.060519 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-2-root/instance:
uuid: "b22b3ca1a8dc463289cfa6d651088845"
format_stamp: "Formatted at 2026-11-16 08:58:43 on dist-test-slave-qt0z"
I20261116 08:58:43.065131 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20261116 08:58:43.067952 22672 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:43.068816 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261116 08:58:43.069080 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-2-root
uuid: "b22b3ca1a8dc463289cfa6d651088845"
format_stamp: "Formatted at 2026-11-16 08:58:43 on dist-test-slave-qt0z"
I20261116 08:58:43.069317 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:43.097699 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:43.098667 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:43.099912 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:43.101998 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:43.102167 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:43.102435 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:43.102578 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:43.137252 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.3:33117
I20261116 08:58:43.137337 22734 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.3:33117 every 8 connection(s)
I20261116 08:58:43.155912 22735 heartbeater.cc:345] Connected to a master server at 127.18.102.62:42867
I20261116 08:58:43.156311 22735 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:43.156961 22735 heartbeater.cc:508] Master 127.18.102.62:42867 requested a full tablet report, sending...
I20261116 08:58:43.158679 22459 ts_manager.cc:210] Registered new tserver with Master: b22b3ca1a8dc463289cfa6d651088845 (127.18.102.3:33117)
I20261116 08:58:43.159176 18840 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018962335s
I20261116 08:58:43.159942 22459 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57768
I20261116 08:58:43.195690 22459 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57774:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261116 08:58:43.244081 22700 tablet_service.cc:1500] Processing CreateTablet for tablet c105a95f83db44128bc8e54aec0c4a36 (DEFAULT_TABLE table=test-workload [id=223cdbb01ebb47a2a25e9e276d07862d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:43.245395 22700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c105a95f83db44128bc8e54aec0c4a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:43.247069 22551 tablet_service.cc:1500] Processing CreateTablet for tablet c105a95f83db44128bc8e54aec0c4a36 (DEFAULT_TABLE table=test-workload [id=223cdbb01ebb47a2a25e9e276d07862d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:43.247642 22626 tablet_service.cc:1500] Processing CreateTablet for tablet c105a95f83db44128bc8e54aec0c4a36 (DEFAULT_TABLE table=test-workload [id=223cdbb01ebb47a2a25e9e276d07862d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:43.248277 22551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c105a95f83db44128bc8e54aec0c4a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:43.248745 22626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c105a95f83db44128bc8e54aec0c4a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:43.258900 22755 tablet_bootstrap.cc:492] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845: Bootstrap starting.
I20261116 08:58:43.263926 22756 tablet_bootstrap.cc:492] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Bootstrap starting.
I20261116 08:58:43.264434 22755 tablet_bootstrap.cc:654] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:43.269037 22757 tablet_bootstrap.cc:492] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec: Bootstrap starting.
I20261116 08:58:43.271060 22756 tablet_bootstrap.cc:654] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:43.271843 22755 tablet_bootstrap.cc:492] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845: No bootstrap required, opened a new log
I20261116 08:58:43.272305 22755 ts_tablet_manager.cc:1449] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20261116 08:58:43.274521 22755 raft_consensus.cc:360] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } }
I20261116 08:58:43.275282 22755 raft_consensus.cc:386] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:43.275529 22755 raft_consensus.cc:748] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22b3ca1a8dc463289cfa6d651088845, State: Initialized, Role: FOLLOWER
I20261116 08:58:43.276037 22757 tablet_bootstrap.cc:654] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:43.276132 22755 consensus_queue.cc:261] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } }
I20261116 08:58:43.276578 22756 tablet_bootstrap.cc:492] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: No bootstrap required, opened a new log
I20261116 08:58:43.277187 22756 ts_tablet_manager.cc:1449] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20261116 08:58:43.280823 22756 raft_consensus.cc:360] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } }
I20261116 08:58:43.281641 22756 raft_consensus.cc:386] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:43.281924 22756 raft_consensus.cc:748] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4f754a75d249769ec22f66b514001c, State: Initialized, Role: FOLLOWER
I20261116 08:58:43.282338 22735 heartbeater.cc:500] Master 127.18.102.62:42867 was elected leader, sending a full tablet report...
I20261116 08:58:43.282606 22755 ts_tablet_manager.cc:1480] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845: Time spent starting tablet: real 0.010s user 0.002s sys 0.008s
I20261116 08:58:43.282609 22756 consensus_queue.cc:261] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } }
I20261116 08:58:43.286197 22661 heartbeater.cc:500] Master 127.18.102.62:42867 was elected leader, sending a full tablet report...
I20261116 08:58:43.286692 22757 tablet_bootstrap.cc:492] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec: No bootstrap required, opened a new log
I20261116 08:58:43.286903 22756 ts_tablet_manager.cc:1480] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20261116 08:58:43.287129 22757 ts_tablet_manager.cc:1449] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20261116 08:58:43.289997 22757 raft_consensus.cc:360] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } }
I20261116 08:58:43.290573 22757 raft_consensus.cc:386] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:43.290807 22757 raft_consensus.cc:748] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e7664954cb4fa087e0575a1d1c9fec, State: Initialized, Role: FOLLOWER
I20261116 08:58:43.291450 22757 consensus_queue.cc:261] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } }
I20261116 08:58:43.296550 22586 heartbeater.cc:500] Master 127.18.102.62:42867 was elected leader, sending a full tablet report...
I20261116 08:58:43.297087 22757 ts_tablet_manager.cc:1480] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20261116 08:58:43.318132 22763 raft_consensus.cc:499] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:43.318552 22763 raft_consensus.cc:521] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } }
I20261116 08:58:43.320688 22763 leader_election.cc:290] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b22b3ca1a8dc463289cfa6d651088845 (127.18.102.3:33117), bd4f754a75d249769ec22f66b514001c (127.18.102.2:43391)
I20261116 08:58:43.328733 22710 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c105a95f83db44128bc8e54aec0c4a36" candidate_uuid: "43e7664954cb4fa087e0575a1d1c9fec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22b3ca1a8dc463289cfa6d651088845" is_pre_election: true
I20261116 08:58:43.329545 22710 raft_consensus.cc:2490] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e7664954cb4fa087e0575a1d1c9fec in term 0.
I20261116 08:58:43.329865 22636 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c105a95f83db44128bc8e54aec0c4a36" candidate_uuid: "43e7664954cb4fa087e0575a1d1c9fec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd4f754a75d249769ec22f66b514001c" is_pre_election: true
I20261116 08:58:43.330518 22636 raft_consensus.cc:2490] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e7664954cb4fa087e0575a1d1c9fec in term 0.
I20261116 08:58:43.330845 22526 leader_election.cc:304] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e7664954cb4fa087e0575a1d1c9fec, b22b3ca1a8dc463289cfa6d651088845; no voters:
I20261116 08:58:43.331785 22763 raft_consensus.cc:2826] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 08:58:43.332154 22763 raft_consensus.cc:499] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 08:58:43.332484 22763 raft_consensus.cc:3096] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:43.337066 22763 raft_consensus.cc:521] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } }
I20261116 08:58:43.338420 22763 leader_election.cc:290] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [CANDIDATE]: Term 1 election: Requested vote from peers b22b3ca1a8dc463289cfa6d651088845 (127.18.102.3:33117), bd4f754a75d249769ec22f66b514001c (127.18.102.2:43391)
I20261116 08:58:43.339234 22710 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c105a95f83db44128bc8e54aec0c4a36" candidate_uuid: "43e7664954cb4fa087e0575a1d1c9fec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22b3ca1a8dc463289cfa6d651088845"
I20261116 08:58:43.339344 22636 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c105a95f83db44128bc8e54aec0c4a36" candidate_uuid: "43e7664954cb4fa087e0575a1d1c9fec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd4f754a75d249769ec22f66b514001c"
I20261116 08:58:43.339802 22710 raft_consensus.cc:3096] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:43.339833 22636 raft_consensus.cc:3096] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:43.346498 22636 raft_consensus.cc:2490] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e7664954cb4fa087e0575a1d1c9fec in term 1.
I20261116 08:58:43.346498 22710 raft_consensus.cc:2490] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e7664954cb4fa087e0575a1d1c9fec in term 1.
I20261116 08:58:43.347735 22526 leader_election.cc:304] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43e7664954cb4fa087e0575a1d1c9fec, b22b3ca1a8dc463289cfa6d651088845, bd4f754a75d249769ec22f66b514001c; no voters:
I20261116 08:58:43.348418 22763 raft_consensus.cc:2826] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:43.349218 22763 raft_consensus.cc:705] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 1 LEADER]: Becoming Leader. State: Replica: 43e7664954cb4fa087e0575a1d1c9fec, State: Running, Role: LEADER
I20261116 08:58:43.350028 22763 consensus_queue.cc:238] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } }
I20261116 08:58:43.356110 22763 consensus_queue.cc:1049] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 08:58:43.357584 22457 catalog_manager.cc:5840] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e7664954cb4fa087e0575a1d1c9fec (127.18.102.1). New cstate: current_term: 1 leader_uuid: "43e7664954cb4fa087e0575a1d1c9fec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } } }
I20261116 08:58:43.358703 22765 consensus_queue.cc:1049] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 08:58:43.615685 22551 tablet_service.cc:1456] Tablet server has 1 leaders and 0 scanners
I20261116 08:58:43.616504 22700 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:43.617748 22626 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
51add35177144b1181d225e6cd8c3a9e | 127.18.102.62:42867 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
43e7664954cb4fa087e0575a1d1c9fec | 127.18.102.1:42633 | HEALTHY | <none> | 1 | 0
b22b3ca1a8dc463289cfa6d651088845 | 127.18.102.3:33117 | HEALTHY | <none> | 0 | 0
bd4f754a75d249769ec22f66b514001c | 127.18.102.2:43391 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20261116 08:58:43.844353 18840 log_verifier.cc:126] Checking tablet c105a95f83db44128bc8e54aec0c4a36
I20261116 08:58:43.907276 18840 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c105a95f83db44128bc8e54aec0c4a36
I20261116 08:58:43.907895 18840 tablet_replica.cc:333] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: stopping tablet replica
I20261116 08:58:43.908440 18840 raft_consensus.cc:2261] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 1 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:43.908910 18840 raft_consensus.cc:2290] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:44.108915 22551 tablet_service.cc:1456] Tablet server has 1 leaders and 0 scanners
I20261116 08:58:44.111743 22700 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:44.111873 22626 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
W20261116 08:58:44.296427 22525 consensus_peers.cc:597] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec -> Peer bd4f754a75d249769ec22f66b514001c (127.18.102.2:43391): Couldn't send request to peer bd4f754a75d249769ec22f66b514001c. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:44.297381 22767 raft_consensus.cc:1022] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec: Attempting to remove follower bd4f754a75d249769ec22f66b514001c from the Raft config. Reason: The tablet replica hosted on peer bd4f754a75d249769ec22f66b514001c has failed
I20261116 08:58:44.299037 22767 consensus_queue.cc:238] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } }
I20261116 08:58:44.302289 22710 raft_consensus.cc:1285] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [term 1 FOLLOWER]: Refusing update from remote peer 43e7664954cb4fa087e0575a1d1c9fec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20261116 08:58:44.303435 22822 consensus_queue.cc:1049] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261116 08:58:44.308501 22767 raft_consensus.cc:2983] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bd4f754a75d249769ec22f66b514001c (127.18.102.2) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } } }
I20261116 08:58:44.310030 22710 raft_consensus.cc:2983] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bd4f754a75d249769ec22f66b514001c (127.18.102.2) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
51add35177144b1181d225e6cd8c3a9e | 127.18.102.62:42867 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
43e7664954cb4fa087e0575a1d1c9fec | 127.18.102.1:42633 | HEALTHY | <none> | 1 | 0
b22b3ca1a8dc463289cfa6d651088845 | 127.18.102.3:33117 | HEALTHY | <none> | 0 | 0
bd4f754a75d249769ec22f66b514001c | 127.18.102.2:43391 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet c105a95f83db44128bc8e54aec0c4a36 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
b22b3ca1a8dc463289cfa6d651088845 (127.18.102.3:33117): RUNNING
bd4f754a75d249769ec22f66b514001c (127.18.102.2:43391): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
43e7664954cb4fa087e0575a1d1c9fec (127.18.102.1:42633): RUNNING [LEADER]
All reported replicas are:
A = b22b3ca1a8dc463289cfa6d651088845
B = bd4f754a75d249769ec22f66b514001c
C = 43e7664954cb4fa087e0575a1d1c9fec
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
I20261116 08:58:44.318461 22458 catalog_manager.cc:5840] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec reported cstate change: config changed from index -1 to 2, VOTER bd4f754a75d249769ec22f66b514001c (127.18.102.2) evicted. New cstate: current_term: 1 leader_uuid: "43e7664954cb4fa087e0575a1d1c9fec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } } }
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20261116 08:58:44.328606 18840 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261116 08:58:44.333947 22626 tablet_service.cc:1547] Processing DeleteTablet for tablet c105a95f83db44128bc8e54aec0c4a36 with delete_type TABLET_DATA_TOMBSTONED (TS bd4f754a75d249769ec22f66b514001c not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:38056
I20261116 08:58:44.336046 22827 ts_tablet_manager.cc:1962] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20261116 08:58:44.336855 22561 consensus_queue.cc:238] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } }
I20261116 08:58:44.341509 22710 raft_consensus.cc:1285] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [term 1 FOLLOWER]: Refusing update from remote peer 43e7664954cb4fa087e0575a1d1c9fec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20261116 08:58:44.341709 22525 consensus_peers.cc:597] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec -> Peer bd4f754a75d249769ec22f66b514001c (127.18.102.2:43391): Couldn't send request to peer bd4f754a75d249769ec22f66b514001c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:44.342808 22822 consensus_queue.cc:1049] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261116 08:58:44.347813 22827 ts_tablet_manager.cc:1975] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20261116 08:58:44.348302 22827 log.cc:1199] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Deleting WAL directory at /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1794819469408847-18840-0/minicluster-data/ts-1-root/wals/c105a95f83db44128bc8e54aec0c4a36
I20261116 08:58:44.347900 22767 raft_consensus.cc:2983] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bd4f754a75d249769ec22f66b514001c (127.18.102.2) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } }
I20261116 08:58:44.349246 22710 raft_consensus.cc:2983] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bd4f754a75d249769ec22f66b514001c (127.18.102.2) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } }
I20261116 08:58:44.349946 22444 catalog_manager.cc:5065] TS bd4f754a75d249769ec22f66b514001c (127.18.102.2:43391): tablet c105a95f83db44128bc8e54aec0c4a36 (table test-workload [id=223cdbb01ebb47a2a25e9e276d07862d]) successfully deleted
I20261116 08:58:44.355369 22445 catalog_manager.cc:5247] ChangeConfig:ADD_PEER:VOTER RPC for tablet c105a95f83db44128bc8e54aec0c4a36 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20261116 08:58:44.358532 22458 catalog_manager.cc:5840] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 reported cstate change: config changed from index 2 to 3, VOTER bd4f754a75d249769ec22f66b514001c (127.18.102.2) added. New cstate: current_term: 1 leader_uuid: "43e7664954cb4fa087e0575a1d1c9fec" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } } }
I20261116 08:58:44.694473 22700 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:44.694963 22551 tablet_service.cc:1456] Tablet server has 1 leaders and 0 scanners
I20261116 08:58:44.703495 22626 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:44.833746 22855 ts_tablet_manager.cc:979] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Initiating tablet copy from peer 43e7664954cb4fa087e0575a1d1c9fec (127.18.102.1:42633)
I20261116 08:58:44.848024 22855 tablet_copy_client.cc:323] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.102.1:42633
I20261116 08:58:44.866703 22571 tablet_copy_service.cc:140] P 43e7664954cb4fa087e0575a1d1c9fec: Received BeginTabletCopySession request for tablet c105a95f83db44128bc8e54aec0c4a36 from peer bd4f754a75d249769ec22f66b514001c ({username='slave'} at 127.0.0.1:43620)
I20261116 08:58:44.867424 22571 tablet_copy_service.cc:161] P 43e7664954cb4fa087e0575a1d1c9fec: Beginning new tablet copy session on tablet c105a95f83db44128bc8e54aec0c4a36 from peer bd4f754a75d249769ec22f66b514001c at {username='slave'} at 127.0.0.1:43620: session id = bd4f754a75d249769ec22f66b514001c-c105a95f83db44128bc8e54aec0c4a36
I20261116 08:58:44.875337 22571 tablet_copy_source_session.cc:215] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec: Tablet Copy: opened 0 blocks and 1 log segments
I20261116 08:58:44.877703 22855 ts_tablet_manager.cc:1962] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20261116 08:58:44.888083 22855 ts_tablet_manager.cc:1975] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20261116 08:58:44.888643 22855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c105a95f83db44128bc8e54aec0c4a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:44.893656 22855 tablet_copy_client.cc:806] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: tablet copy: Starting download of 0 data blocks...
I20261116 08:58:44.894155 22855 tablet_copy_client.cc:670] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: tablet copy: Starting download of 1 WAL segments...
I20261116 08:58:44.897574 22855 tablet_copy_client.cc:538] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261116 08:58:44.902879 22855 tablet_bootstrap.cc:492] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Bootstrap starting.
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
51add35177144b1181d225e6cd8c3a9e | 127.18.102.62:42867 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
43e7664954cb4fa087e0575a1d1c9fec | 127.18.102.1:42633 | HEALTHY | <none> | 1 | 0
b22b3ca1a8dc463289cfa6d651088845 | 127.18.102.3:33117 | HEALTHY | <none> | 0 | 0
bd4f754a75d249769ec22f66b514001c | 127.18.102.2:43391 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
I20261116 08:58:44.919991 22855 tablet_bootstrap.cc:492] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
I20261116 08:58:44.920787 22855 tablet_bootstrap.cc:492] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Bootstrap complete.
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet c105a95f83db44128bc8e54aec0c4a36 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
b22b3ca1a8dc463289cfa6d651088845 (127.18.102.3:33117): RUNNING
43e7664954cb4fa087e0575a1d1c9fec (127.18.102.1:42633): RUNNING [LEADER]
bd4f754a75d249769ec22f66b514001c (127.18.102.2:43391): not running
I20261116 08:58:44.921429 22855 ts_tablet_manager.cc:1449] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = b22b3ca1a8dc463289cfa6d651088845
B = 43e7664954cb4fa087e0575a1d1c9fec
C = bd4f754a75d249769ec22f66b514001c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 3 | Yes
B | A B* C | 1 | 3 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20261116 08:58:44.924291 22855 raft_consensus.cc:360] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } }
I20261116 08:58:44.925137 22855 raft_consensus.cc:748] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4f754a75d249769ec22f66b514001c, State: Initialized, Role: FOLLOWER
I20261116 08:58:44.925882 22855 consensus_queue.cc:261] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b22b3ca1a8dc463289cfa6d651088845" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 33117 } } peers { permanent_uuid: "43e7664954cb4fa087e0575a1d1c9fec" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 42633 } } peers { permanent_uuid: "bd4f754a75d249769ec22f66b514001c" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 43391 } }
I20261116 08:58:44.928488 18840 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261116 08:58:44.930915 22855 ts_tablet_manager.cc:1480] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20261116 08:58:44.932603 22571 tablet_copy_service.cc:342] P 43e7664954cb4fa087e0575a1d1c9fec: Request end of tablet copy session bd4f754a75d249769ec22f66b514001c-c105a95f83db44128bc8e54aec0c4a36 received from {username='slave'} at 127.0.0.1:43620
I20261116 08:58:44.932910 22571 tablet_copy_service.cc:434] P 43e7664954cb4fa087e0575a1d1c9fec: ending tablet copy session bd4f754a75d249769ec22f66b514001c-c105a95f83db44128bc8e54aec0c4a36 on tablet c105a95f83db44128bc8e54aec0c4a36 with peer bd4f754a75d249769ec22f66b514001c
I20261116 08:58:45.249349 22636 raft_consensus.cc:1227] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20261116 08:58:45.370831 22551 tablet_service.cc:1456] Tablet server has 1 leaders and 0 scanners
I20261116 08:58:45.385457 22700 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:45.385545 22626 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
51add35177144b1181d225e6cd8c3a9e | 127.18.102.62:42867 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
43e7664954cb4fa087e0575a1d1c9fec | 127.18.102.1:42633 | HEALTHY | <none> | 1 | 0
b22b3ca1a8dc463289cfa6d651088845 | 127.18.102.3:33117 | HEALTHY | <none> | 0 | 0
bd4f754a75d249769ec22f66b514001c | 127.18.102.2:43391 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20261116 08:58:45.578102 18840 log_verifier.cc:126] Checking tablet c105a95f83db44128bc8e54aec0c4a36
I20261116 08:58:45.629642 18840 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c105a95f83db44128bc8e54aec0c4a36
I20261116 08:58:45.631794 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
I20261116 08:58:45.650385 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:45.650921 18840 tablet_replica.cc:333] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec: stopping tablet replica
I20261116 08:58:45.651393 18840 raft_consensus.cc:2261] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:45.652185 18840 raft_consensus.cc:2290] T c105a95f83db44128bc8e54aec0c4a36 P 43e7664954cb4fa087e0575a1d1c9fec [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:45.671702 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
I20261116 08:58:45.680452 18840 tablet_server.cc:179] TabletServer@127.18.102.2:0 shutting down...
I20261116 08:58:45.696928 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:45.697496 18840 tablet_replica.cc:333] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c: stopping tablet replica
I20261116 08:58:45.697983 18840 raft_consensus.cc:2261] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 1 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:45.698362 18840 raft_consensus.cc:2290] T c105a95f83db44128bc8e54aec0c4a36 P bd4f754a75d249769ec22f66b514001c [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:45.716830 18840 tablet_server.cc:196] TabletServer@127.18.102.2:0 shutdown complete.
I20261116 08:58:45.725524 18840 tablet_server.cc:179] TabletServer@127.18.102.3:0 shutting down...
I20261116 08:58:45.743975 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:45.744560 18840 tablet_replica.cc:333] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845: stopping tablet replica
I20261116 08:58:45.745042 18840 raft_consensus.cc:2261] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:45.745498 18840 raft_consensus.cc:2290] T c105a95f83db44128bc8e54aec0c4a36 P b22b3ca1a8dc463289cfa6d651088845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:45.765054 18840 tablet_server.cc:196] TabletServer@127.18.102.3:0 shutdown complete.
I20261116 08:58:45.773788 18840 master.cc:585] Master@127.18.102.62:42867 shutting down...
I20261116 08:58:45.789290 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:45.789773 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:45.790068 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P 51add35177144b1181d225e6cd8c3a9e: stopping tablet replica
I20261116 08:58:45.807888 18840 master.cc:607] Master@127.18.102.62:42867 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (3147 ms)
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/1
I20261116 08:58:45.833484 18840 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.102.62:40211
I20261116 08:58:45.834468 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:45.839200 22890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:45.840081 22891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:45.841617 22893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:45.841905 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:45.842788 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:45.842972 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:45.843142 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819525843105 us; error 0 us; skew 500 ppm
I20261116 08:58:45.843698 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:45.845790 18840 webserver.cc:577] Webserver started at http://127.18.102.62:33271/ using document root <none> and password file <none>
I20261116 08:58:45.846253 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:45.846428 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:45.846719 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:45.847885 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/master-0-root/instance:
uuid: "d45ca5c653ae4ed78f38ddb509c4aaea"
format_stamp: "Formatted at 2026-11-16 08:58:45 on dist-test-slave-qt0z"
I20261116 08:58:45.852061 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20261116 08:58:45.855098 22898 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:45.855856 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261116 08:58:45.856139 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/master-0-root
uuid: "d45ca5c653ae4ed78f38ddb509c4aaea"
format_stamp: "Formatted at 2026-11-16 08:58:45 on dist-test-slave-qt0z"
I20261116 08:58:45.856400 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:45.872048 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:45.872956 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:45.905191 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.62:40211
I20261116 08:58:45.905283 22949 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.62:40211 every 8 connection(s)
I20261116 08:58:45.908783 22950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:45.918933 22950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea: Bootstrap starting.
I20261116 08:58:45.923043 22950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:45.926741 22950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea: No bootstrap required, opened a new log
I20261116 08:58:45.928658 22950 raft_consensus.cc:360] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45ca5c653ae4ed78f38ddb509c4aaea" member_type: VOTER }
I20261116 08:58:45.928992 22950 raft_consensus.cc:386] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:45.929175 22950 raft_consensus.cc:748] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45ca5c653ae4ed78f38ddb509c4aaea, State: Initialized, Role: FOLLOWER
I20261116 08:58:45.929636 22950 consensus_queue.cc:261] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45ca5c653ae4ed78f38ddb509c4aaea" member_type: VOTER }
I20261116 08:58:45.930011 22950 raft_consensus.cc:400] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 08:58:45.930195 22950 raft_consensus.cc:499] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 08:58:45.930392 22950 raft_consensus.cc:3096] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:45.934574 22950 raft_consensus.cc:521] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45ca5c653ae4ed78f38ddb509c4aaea" member_type: VOTER }
I20261116 08:58:45.934993 22950 leader_election.cc:304] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d45ca5c653ae4ed78f38ddb509c4aaea; no voters:
I20261116 08:58:45.936030 22950 leader_election.cc:290] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 08:58:45.936342 22953 raft_consensus.cc:2826] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:45.937590 22953 raft_consensus.cc:705] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [term 1 LEADER]: Becoming Leader. State: Replica: d45ca5c653ae4ed78f38ddb509c4aaea, State: Running, Role: LEADER
I20261116 08:58:45.938230 22953 consensus_queue.cc:238] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45ca5c653ae4ed78f38ddb509c4aaea" member_type: VOTER }
I20261116 08:58:45.938798 22950 sys_catalog.cc:565] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [sys.catalog]: configured and running, proceeding with master startup.
I20261116 08:58:45.942194 22954 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d45ca5c653ae4ed78f38ddb509c4aaea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45ca5c653ae4ed78f38ddb509c4aaea" member_type: VOTER } }
I20261116 08:58:45.942360 22955 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [sys.catalog]: SysCatalogTable state changed. Reason: New leader d45ca5c653ae4ed78f38ddb509c4aaea. Latest consensus state: current_term: 1 leader_uuid: "d45ca5c653ae4ed78f38ddb509c4aaea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45ca5c653ae4ed78f38ddb509c4aaea" member_type: VOTER } }
I20261116 08:58:45.942977 22954 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:45.943061 22955 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [sys.catalog]: This master's current role is: LEADER
I20261116 08:58:45.945664 22960 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 08:58:45.950074 22960 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 08:58:45.954972 18840 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261116 08:58:45.958271 22960 catalog_manager.cc:1398] Generated new cluster ID: 10867adae4e34fa1a3b35f253ff3e4f8
I20261116 08:58:45.958539 22960 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 08:58:45.992033 22960 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 08:58:45.993218 22960 catalog_manager.cc:1555] Loading token signing keys...
I20261116 08:58:46.006242 22960 catalog_manager.cc:6221] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea: Generated new TSK 0
I20261116 08:58:46.006810 22960 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 08:58:46.021088 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:46.026499 22971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:46.027782 22972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:46.029305 22974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:46.029716 18840 server_base.cc:1061] running on GCE node
I20261116 08:58:46.030529 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:46.030709 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:46.030879 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819526030841 us; error 0 us; skew 500 ppm
I20261116 08:58:46.031456 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:46.033586 18840 webserver.cc:577] Webserver started at http://127.18.102.1:39453/ using document root <none> and password file <none>
I20261116 08:58:46.034050 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:46.034238 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:46.034509 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:46.035631 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-0-root/instance:
uuid: "0eef0132476a4eb09475312095772337"
format_stamp: "Formatted at 2026-11-16 08:58:46 on dist-test-slave-qt0z"
I20261116 08:58:46.039734 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20261116 08:58:46.042723 22979 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:46.043495 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261116 08:58:46.043800 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-0-root
uuid: "0eef0132476a4eb09475312095772337"
format_stamp: "Formatted at 2026-11-16 08:58:46 on dist-test-slave-qt0z"
I20261116 08:58:46.044083 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:46.055356 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:46.056409 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:46.057749 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:46.059818 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:46.060022 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:46.060257 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:46.060420 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:46.095692 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.1:35919
I20261116 08:58:46.095783 23041 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.1:35919 every 8 connection(s)
I20261116 08:58:46.100040 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:46.106885 23046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:46.108881 23047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:46.111402 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:46.112351 23049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:46.112574 23042 heartbeater.cc:345] Connected to a master server at 127.18.102.62:40211
I20261116 08:58:46.113061 23042 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:46.113384 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:46.113657 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:46.113854 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819526113835 us; error 0 us; skew 500 ppm
I20261116 08:58:46.113899 23042 heartbeater.cc:508] Master 127.18.102.62:40211 requested a full tablet report, sending...
I20261116 08:58:46.114655 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:46.116284 22915 ts_manager.cc:210] Registered new tserver with Master: 0eef0132476a4eb09475312095772337 (127.18.102.1:35919)
I20261116 08:58:46.117304 18840 webserver.cc:577] Webserver started at http://127.18.102.2:39351/ using document root <none> and password file <none>
I20261116 08:58:46.117920 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:46.118172 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:46.118469 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:46.118580 22915 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56726
I20261116 08:58:46.119889 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-1-root/instance:
uuid: "c6acbb08cfd14f12bb2c18a08e498b78"
format_stamp: "Formatted at 2026-11-16 08:58:46 on dist-test-slave-qt0z"
I20261116 08:58:46.124343 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20261116 08:58:46.127439 23054 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:46.128167 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261116 08:58:46.128437 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-1-root
uuid: "c6acbb08cfd14f12bb2c18a08e498b78"
format_stamp: "Formatted at 2026-11-16 08:58:46 on dist-test-slave-qt0z"
I20261116 08:58:46.128713 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:46.142128 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:46.143031 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:46.144382 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:46.146522 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:46.146740 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:46.146958 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:46.147138 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:46.184506 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.2:41687
I20261116 08:58:46.184593 23116 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.2:41687 every 8 connection(s)
I20261116 08:58:46.189057 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:46.195456 23120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:46.196045 23121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:46.198671 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:46.200035 23124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:46.200569 23117 heartbeater.cc:345] Connected to a master server at 127.18.102.62:40211
I20261116 08:58:46.200886 23117 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:46.201442 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20261116 08:58:46.201515 23117 heartbeater.cc:508] Master 127.18.102.62:40211 requested a full tablet report, sending...
W20261116 08:58:46.201704 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:46.201977 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819526201959 us; error 0 us; skew 500 ppm
I20261116 08:58:46.202535 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:46.203734 22915 ts_manager.cc:210] Registered new tserver with Master: c6acbb08cfd14f12bb2c18a08e498b78 (127.18.102.2:41687)
I20261116 08:58:46.204912 18840 webserver.cc:577] Webserver started at http://127.18.102.3:42649/ using document root <none> and password file <none>
I20261116 08:58:46.205291 22915 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56732
I20261116 08:58:46.205406 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:46.205725 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:46.206001 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:46.207242 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-2-root/instance:
uuid: "13be8ace710348bca7c547205274be5e"
format_stamp: "Formatted at 2026-11-16 08:58:46 on dist-test-slave-qt0z"
I20261116 08:58:46.211362 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20261116 08:58:46.214354 23128 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:46.215095 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261116 08:58:46.215433 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-2-root
uuid: "13be8ace710348bca7c547205274be5e"
format_stamp: "Formatted at 2026-11-16 08:58:46 on dist-test-slave-qt0z"
I20261116 08:58:46.215729 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:46.234331 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:46.235410 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:46.236740 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:46.238771 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:46.238976 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:46.239230 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:46.239398 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:46.275319 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.3:45355
I20261116 08:58:46.275395 23190 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.3:45355 every 8 connection(s)
I20261116 08:58:46.279956 18840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 08:58:46.286437 23194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 08:58:46.287209 23195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:46.289324 18840 server_base.cc:1061] running on GCE node
W20261116 08:58:46.289981 23197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 08:58:46.290983 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261116 08:58:46.291302 18840 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261116 08:58:46.291524 18840 hybrid_clock.cc:648] HybridClock initialized: now 1794819526291506 us; error 0 us; skew 500 ppm
I20261116 08:58:46.291870 23191 heartbeater.cc:345] Connected to a master server at 127.18.102.62:40211
I20261116 08:58:46.292192 18840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261116 08:58:46.292249 23191 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:46.293190 23191 heartbeater.cc:508] Master 127.18.102.62:40211 requested a full tablet report, sending...
I20261116 08:58:46.294735 18840 webserver.cc:577] Webserver started at http://127.18.102.4:33415/ using document root <none> and password file <none>
I20261116 08:58:46.295354 18840 fs_manager.cc:362] Metadata directory not provided
I20261116 08:58:46.295405 22915 ts_manager.cc:210] Registered new tserver with Master: 13be8ace710348bca7c547205274be5e (127.18.102.3:45355)
I20261116 08:58:46.295639 18840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 08:58:46.296015 18840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 08:58:46.296839 22915 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56746
I20261116 08:58:46.297040 18840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-3-root/instance:
uuid: "c5a51f9f9ba246f7961b8fc5e4734f92"
format_stamp: "Formatted at 2026-11-16 08:58:46 on dist-test-slave-qt0z"
I20261116 08:58:46.301389 18840 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20261116 08:58:46.304396 23202 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:46.305155 18840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261116 08:58:46.305434 18840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-3-root
uuid: "c5a51f9f9ba246f7961b8fc5e4734f92"
format_stamp: "Formatted at 2026-11-16 08:58:46 on dist-test-slave-qt0z"
I20261116 08:58:46.305720 18840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 08:58:46.326574 18840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 08:58:46.327699 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 08:58:46.329022 18840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 08:58:46.331499 18840 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 08:58:46.331718 18840 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:46.332007 18840 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 08:58:46.332182 18840 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 08:58:46.369455 18840 rpc_server.cc:307] RPC server started. Bound to: 127.18.102.4:40663
I20261116 08:58:46.369544 23264 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.102.4:40663 every 8 connection(s)
I20261116 08:58:46.382148 23265 heartbeater.cc:345] Connected to a master server at 127.18.102.62:40211
I20261116 08:58:46.382534 23265 heartbeater.cc:462] Registering TS with master...
I20261116 08:58:46.383167 23265 heartbeater.cc:508] Master 127.18.102.62:40211 requested a full tablet report, sending...
I20261116 08:58:46.384948 22915 ts_manager.cc:210] Registered new tserver with Master: c5a51f9f9ba246f7961b8fc5e4734f92 (127.18.102.4:40663)
I20261116 08:58:46.386077 18840 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.013771245s
I20261116 08:58:46.386116 22915 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56752
I20261116 08:58:46.422302 22915 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56766:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261116 08:58:46.460325 23230 tablet_service.cc:1500] Processing CreateTablet for tablet 2bbeb51ea6804e9b9c9be801ff344029 (DEFAULT_TABLE table=test-workload [id=645d27f62f294744be1e0b0f806c95cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:46.461541 23230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bbeb51ea6804e9b9c9be801ff344029. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:46.466552 23007 tablet_service.cc:1500] Processing CreateTablet for tablet 2bbeb51ea6804e9b9c9be801ff344029 (DEFAULT_TABLE table=test-workload [id=645d27f62f294744be1e0b0f806c95cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:46.467798 23007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bbeb51ea6804e9b9c9be801ff344029. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:46.472908 23082 tablet_service.cc:1500] Processing CreateTablet for tablet 2bbeb51ea6804e9b9c9be801ff344029 (DEFAULT_TABLE table=test-workload [id=645d27f62f294744be1e0b0f806c95cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 08:58:46.474161 23082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bbeb51ea6804e9b9c9be801ff344029. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:46.477862 23285 tablet_bootstrap.cc:492] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92: Bootstrap starting.
I20261116 08:58:46.481814 23286 tablet_bootstrap.cc:492] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337: Bootstrap starting.
I20261116 08:58:46.485503 23285 tablet_bootstrap.cc:654] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:46.488529 23286 tablet_bootstrap.cc:654] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:46.489173 23288 tablet_bootstrap.cc:492] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78: Bootstrap starting.
I20261116 08:58:46.490293 23285 tablet_bootstrap.cc:492] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92: No bootstrap required, opened a new log
I20261116 08:58:46.490832 23285 ts_tablet_manager.cc:1449] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20261116 08:58:46.493940 23285 raft_consensus.cc:360] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92" member_type: VOTER last_known_addr { host: "127.18.102.4" port: 40663 } } peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } }
I20261116 08:58:46.494715 23285 raft_consensus.cc:386] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:46.495012 23285 raft_consensus.cc:748] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a51f9f9ba246f7961b8fc5e4734f92, State: Initialized, Role: FOLLOWER
I20261116 08:58:46.495983 23286 tablet_bootstrap.cc:492] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337: No bootstrap required, opened a new log
I20261116 08:58:46.496248 23288 tablet_bootstrap.cc:654] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78: Neither blocks nor log segments found. Creating new log.
I20261116 08:58:46.495751 23285 consensus_queue.cc:261] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92" member_type: VOTER last_known_addr { host: "127.18.102.4" port: 40663 } } peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } }
I20261116 08:58:46.496379 23286 ts_tablet_manager.cc:1449] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.001s
I20261116 08:58:46.498340 23265 heartbeater.cc:500] Master 127.18.102.62:40211 was elected leader, sending a full tablet report...
I20261116 08:58:46.498780 23285 ts_tablet_manager.cc:1480] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20261116 08:58:46.499594 23286 raft_consensus.cc:360] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92" member_type: VOTER last_known_addr { host: "127.18.102.4" port: 40663 } } peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } }
I20261116 08:58:46.500177 23286 raft_consensus.cc:386] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:46.500499 23286 raft_consensus.cc:748] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eef0132476a4eb09475312095772337, State: Initialized, Role: FOLLOWER
I20261116 08:58:46.501458 23286 consensus_queue.cc:261] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92" member_type: VOTER last_known_addr { host: "127.18.102.4" port: 40663 } } peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } }
I20261116 08:58:46.503010 23288 tablet_bootstrap.cc:492] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78: No bootstrap required, opened a new log
I20261116 08:58:46.503651 23288 ts_tablet_manager.cc:1449] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20261116 08:58:46.505015 23042 heartbeater.cc:500] Master 127.18.102.62:40211 was elected leader, sending a full tablet report...
I20261116 08:58:46.505504 23286 ts_tablet_manager.cc:1480] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20261116 08:58:46.506565 23288 raft_consensus.cc:360] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92" member_type: VOTER last_known_addr { host: "127.18.102.4" port: 40663 } } peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } }
I20261116 08:58:46.507275 23288 raft_consensus.cc:386] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 08:58:46.507601 23288 raft_consensus.cc:748] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6acbb08cfd14f12bb2c18a08e498b78, State: Initialized, Role: FOLLOWER
I20261116 08:58:46.508291 23288 consensus_queue.cc:261] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92" member_type: VOTER last_known_addr { host: "127.18.102.4" port: 40663 } } peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } }
I20261116 08:58:46.510418 23117 heartbeater.cc:500] Master 127.18.102.62:40211 was elected leader, sending a full tablet report...
I20261116 08:58:46.510950 23288 ts_tablet_manager.cc:1480] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20261116 08:58:46.929306 23293 raft_consensus.cc:499] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 08:58:46.929770 23293 raft_consensus.cc:521] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92" member_type: VOTER last_known_addr { host: "127.18.102.4" port: 40663 } } peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } }
I20261116 08:58:46.931823 23293 leader_election.cc:290] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5a51f9f9ba246f7961b8fc5e4734f92 (127.18.102.4:40663), 0eef0132476a4eb09475312095772337 (127.18.102.1:35919)
I20261116 08:58:46.941752 23240 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2bbeb51ea6804e9b9c9be801ff344029" candidate_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92" is_pre_election: true
I20261116 08:58:46.942137 23017 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2bbeb51ea6804e9b9c9be801ff344029" candidate_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eef0132476a4eb09475312095772337" is_pre_election: true
I20261116 08:58:46.942342 23240 raft_consensus.cc:2490] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6acbb08cfd14f12bb2c18a08e498b78 in term 0.
I20261116 08:58:46.942801 23017 raft_consensus.cc:2490] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6acbb08cfd14f12bb2c18a08e498b78 in term 0.
I20261116 08:58:46.943429 23057 leader_election.cc:304] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5a51f9f9ba246f7961b8fc5e4734f92, c6acbb08cfd14f12bb2c18a08e498b78; no voters:
I20261116 08:58:46.944198 23293 raft_consensus.cc:2826] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 08:58:46.944453 23293 raft_consensus.cc:499] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 08:58:46.944667 23293 raft_consensus.cc:3096] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:46.949186 23293 raft_consensus.cc:521] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92" member_type: VOTER last_known_addr { host: "127.18.102.4" port: 40663 } } peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } }
I20261116 08:58:46.950738 23293 leader_election.cc:290] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [CANDIDATE]: Term 1 election: Requested vote from peers c5a51f9f9ba246f7961b8fc5e4734f92 (127.18.102.4:40663), 0eef0132476a4eb09475312095772337 (127.18.102.1:35919)
I20261116 08:58:46.951375 23240 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2bbeb51ea6804e9b9c9be801ff344029" candidate_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92"
I20261116 08:58:46.951663 23017 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2bbeb51ea6804e9b9c9be801ff344029" candidate_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eef0132476a4eb09475312095772337"
I20261116 08:58:46.951831 23240 raft_consensus.cc:3096] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:46.952234 23017 raft_consensus.cc:3096] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 0 FOLLOWER]: Advancing to term 1
I20261116 08:58:46.956432 23240 raft_consensus.cc:2490] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6acbb08cfd14f12bb2c18a08e498b78 in term 1.
I20261116 08:58:46.956650 23017 raft_consensus.cc:2490] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6acbb08cfd14f12bb2c18a08e498b78 in term 1.
I20261116 08:58:46.957412 23057 leader_election.cc:304] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5a51f9f9ba246f7961b8fc5e4734f92, c6acbb08cfd14f12bb2c18a08e498b78; no voters:
I20261116 08:58:46.958199 23293 raft_consensus.cc:2826] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 08:58:46.959999 23293 raft_consensus.cc:705] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 1 LEADER]: Becoming Leader. State: Replica: c6acbb08cfd14f12bb2c18a08e498b78, State: Running, Role: LEADER
I20261116 08:58:46.960673 23293 consensus_queue.cc:238] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92" member_type: VOTER last_known_addr { host: "127.18.102.4" port: 40663 } } peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } }
I20261116 08:58:46.967104 23293 consensus_queue.cc:1049] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 08:58:46.967939 22914 catalog_manager.cc:5840] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6acbb08cfd14f12bb2c18a08e498b78 (127.18.102.2). New cstate: current_term: 1 leader_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92" member_type: VOTER last_known_addr { host: "127.18.102.4" port: 40663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } health_report { overall_health: HEALTHY } } }
I20261116 08:58:46.969005 23293 consensus_queue.cc:1049] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a51f9f9ba246f7961b8fc5e4734f92" member_type: VOTER last_known_addr { host: "127.18.102.4" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 08:58:47.268750 23007 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:47.272030 23156 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:47.280026 23230 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:47.284152 23082 tablet_service.cc:1456] Tablet server has 1 leaders and 0 scanners
I20261116 08:58:47.299518 23191 heartbeater.cc:500] Master 127.18.102.62:40211 was elected leader, sending a full tablet report...
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d45ca5c653ae4ed78f38ddb509c4aaea | 127.18.102.62:40211 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0eef0132476a4eb09475312095772337 | 127.18.102.1:35919 | HEALTHY | <none> | 0 | 0
13be8ace710348bca7c547205274be5e | 127.18.102.3:45355 | HEALTHY | <none> | 0 | 0
c5a51f9f9ba246f7961b8fc5e4734f92 | 127.18.102.4:40663 | HEALTHY | <none> | 0 | 0
c6acbb08cfd14f12bb2c18a08e498b78 | 127.18.102.2:41687 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 13be8ace710348bca7c547205274be5e | 127.18.102.3:45355 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20261116 08:58:47.734297 18840 log_verifier.cc:126] Checking tablet 2bbeb51ea6804e9b9c9be801ff344029
I20261116 08:58:47.794056 18840 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 2bbeb51ea6804e9b9c9be801ff344029
I20261116 08:58:47.794606 18840 tablet_replica.cc:333] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92: stopping tablet replica
I20261116 08:58:47.795233 18840 raft_consensus.cc:2261] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:47.795738 18840 raft_consensus.cc:2290] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:48.016794 23156 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:48.017462 23007 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:48.023820 23230 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:48.027277 23082 tablet_service.cc:1456] Tablet server has 1 leaders and 0 scanners
I20261116 08:58:48.062999 23057 consensus_queue.cc:580] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [LEADER]: The tablet replica hosted on peer c5a51f9f9ba246f7961b8fc5e4734f92 has failed
W20261116 08:58:48.063467 23057 consensus_peers.cc:597] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 -> Peer c5a51f9f9ba246f7961b8fc5e4734f92 (127.18.102.4:40663): Couldn't send request to peer c5a51f9f9ba246f7961b8fc5e4734f92. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:48.090353 23092 consensus_queue.cc:238] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } }
I20261116 08:58:48.110062 23017 raft_consensus.cc:1285] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 1 FOLLOWER]: Refusing update from remote peer c6acbb08cfd14f12bb2c18a08e498b78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20261116 08:58:48.111821 23370 consensus_queue.cc:1049] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261116 08:58:48.135602 23339 raft_consensus.cc:2983] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c5a51f9f9ba246f7961b8fc5e4734f92 (127.18.102.4) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } } }
I20261116 08:58:48.137384 23016 raft_consensus.cc:2983] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c5a51f9f9ba246f7961b8fc5e4734f92 (127.18.102.4) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } } }
I20261116 08:58:48.149379 22915 catalog_manager.cc:5840] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 reported cstate change: config changed from index -1 to 2, VOTER c5a51f9f9ba246f7961b8fc5e4734f92 (127.18.102.4) evicted. New cstate: current_term: 1 leader_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } health_report { overall_health: HEALTHY } } }
I20261116 08:58:48.146883 22899 catalog_manager.cc:5247] ChangeConfig:REMOVE_PEER RPC for tablet 2bbeb51ea6804e9b9c9be801ff344029 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20261116 08:58:48.204898 23092 consensus_queue.cc:238] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } } peers { permanent_uuid: "13be8ace710348bca7c547205274be5e" member_type: NON_VOTER last_known_addr { host: "127.18.102.3" port: 45355 } attrs { promote: true } }
I20261116 08:58:48.208760 23230 tablet_service.cc:1547] Processing DeleteTablet for tablet 2bbeb51ea6804e9b9c9be801ff344029 with delete_type TABLET_DATA_TOMBSTONED (TS c5a51f9f9ba246f7961b8fc5e4734f92 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:54772
I20261116 08:58:48.220052 23378 ts_tablet_manager.cc:1962] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20261116 08:58:48.237794 23016 raft_consensus.cc:1285] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 1 FOLLOWER]: Refusing update from remote peer c6acbb08cfd14f12bb2c18a08e498b78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20261116 08:58:48.239495 23339 consensus_queue.cc:1049] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261116 08:58:48.247020 23339 raft_consensus.cc:2983] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 13be8ace710348bca7c547205274be5e (127.18.102.3) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } } peers { permanent_uuid: "13be8ace710348bca7c547205274be5e" member_type: NON_VOTER last_known_addr { host: "127.18.102.3" port: 45355 } attrs { promote: true } } }
I20261116 08:58:48.250294 23016 raft_consensus.cc:2983] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 13be8ace710348bca7c547205274be5e (127.18.102.3) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } } peers { permanent_uuid: "13be8ace710348bca7c547205274be5e" member_type: NON_VOTER last_known_addr { host: "127.18.102.3" port: 45355 } attrs { promote: true } } }
I20261116 08:58:48.270365 22899 catalog_manager.cc:5247] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2bbeb51ea6804e9b9c9be801ff344029 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20261116 08:58:48.273622 22912 catalog_manager.cc:5840] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 reported cstate change: config changed from index 2 to 3, NON_VOTER 13be8ace710348bca7c547205274be5e (127.18.102.3) added. New cstate: current_term: 1 leader_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } } peers { permanent_uuid: "13be8ace710348bca7c547205274be5e" member_type: NON_VOTER last_known_addr { host: "127.18.102.3" port: 45355 } attrs { promote: true } } }
W20261116 08:58:48.278326 23057 consensus_peers.cc:597] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 -> Peer 13be8ace710348bca7c547205274be5e (127.18.102.3:45355): Couldn't send request to peer 13be8ace710348bca7c547205274be5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bbeb51ea6804e9b9c9be801ff344029. This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:48.280332 23378 ts_tablet_manager.cc:1975] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20261116 08:58:48.280889 23378 log.cc:1199] T 2bbeb51ea6804e9b9c9be801ff344029 P c5a51f9f9ba246f7961b8fc5e4734f92: Deleting WAL directory at /tmp/dist-test-taska4bhhW/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1794819469408847-18840-0/minicluster-data/ts-3-root/wals/2bbeb51ea6804e9b9c9be801ff344029
I20261116 08:58:48.282814 22901 catalog_manager.cc:5065] TS c5a51f9f9ba246f7961b8fc5e4734f92 (127.18.102.4:40663): tablet 2bbeb51ea6804e9b9c9be801ff344029 (table test-workload [id=645d27f62f294744be1e0b0f806c95cc]) successfully deleted
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d45ca5c653ae4ed78f38ddb509c4aaea | 127.18.102.62:40211 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0eef0132476a4eb09475312095772337 | 127.18.102.1:35919 | HEALTHY | <none> | 0 | 0
13be8ace710348bca7c547205274be5e | 127.18.102.3:45355 | HEALTHY | <none> | 0 | 0
c5a51f9f9ba246f7961b8fc5e4734f92 | 127.18.102.4:40663 | HEALTHY | <none> | 0 | 0
c6acbb08cfd14f12bb2c18a08e498b78 | 127.18.102.2:41687 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2bbeb51ea6804e9b9c9be801ff344029 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
c5a51f9f9ba246f7961b8fc5e4734f92 (127.18.102.4:40663): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
0eef0132476a4eb09475312095772337 (127.18.102.1:35919): RUNNING
c6acbb08cfd14f12bb2c18a08e498b78 (127.18.102.2:41687): RUNNING [LEADER]
All reported replicas are:
A = c5a51f9f9ba246f7961b8fc5e4734f92
B = 0eef0132476a4eb09475312095772337
C = c6acbb08cfd14f12bb2c18a08e498b78
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 13be8ace710348bca7c547205274be5e | 127.18.102.3:45355 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20261116 08:58:48.334156 18840 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20261116 08:58:48.515239 23261 debug-util.cc:398] Leaking SignalData structure 0x7b08001aeac0 after lost signal to thread 18843
I20261116 08:58:48.661199 23397 ts_tablet_manager.cc:979] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e: Initiating tablet copy from peer c6acbb08cfd14f12bb2c18a08e498b78 (127.18.102.2:41687)
I20261116 08:58:48.677892 23397 tablet_copy_client.cc:323] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e: tablet copy: Beginning tablet copy session from remote peer at address 127.18.102.2:41687
I20261116 08:58:48.692274 23102 tablet_copy_service.cc:140] P c6acbb08cfd14f12bb2c18a08e498b78: Received BeginTabletCopySession request for tablet 2bbeb51ea6804e9b9c9be801ff344029 from peer 13be8ace710348bca7c547205274be5e ({username='slave'} at 127.0.0.1:42212)
I20261116 08:58:48.692796 23102 tablet_copy_service.cc:161] P c6acbb08cfd14f12bb2c18a08e498b78: Beginning new tablet copy session on tablet 2bbeb51ea6804e9b9c9be801ff344029 from peer 13be8ace710348bca7c547205274be5e at {username='slave'} at 127.0.0.1:42212: session id = 13be8ace710348bca7c547205274be5e-2bbeb51ea6804e9b9c9be801ff344029
I20261116 08:58:48.698362 23102 tablet_copy_source_session.cc:215] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78: Tablet Copy: opened 0 blocks and 1 log segments
I20261116 08:58:48.701452 23397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bbeb51ea6804e9b9c9be801ff344029. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 08:58:48.739271 23397 tablet_copy_client.cc:806] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e: tablet copy: Starting download of 0 data blocks...
I20261116 08:58:48.740010 23397 tablet_copy_client.cc:670] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e: tablet copy: Starting download of 1 WAL segments...
I20261116 08:58:48.742877 23007 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:48.746289 23397 tablet_copy_client.cc:538] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261116 08:58:48.756784 23156 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:48.758002 23397 tablet_bootstrap.cc:492] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e: Bootstrap starting.
I20261116 08:58:48.769853 23082 tablet_service.cc:1456] Tablet server has 1 leaders and 0 scanners
I20261116 08:58:48.788486 23230 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:48.793071 23397 tablet_bootstrap.cc:492] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261116 08:58:48.793980 23397 tablet_bootstrap.cc:492] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e: Bootstrap complete.
I20261116 08:58:48.794574 23397 ts_tablet_manager.cc:1449] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.016s
I20261116 08:58:48.796808 23397 raft_consensus.cc:360] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } } peers { permanent_uuid: "13be8ace710348bca7c547205274be5e" member_type: NON_VOTER last_known_addr { host: "127.18.102.3" port: 45355 } attrs { promote: true } }
I20261116 08:58:48.797391 23397 raft_consensus.cc:748] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13be8ace710348bca7c547205274be5e, State: Initialized, Role: LEARNER
I20261116 08:58:48.797922 23397 consensus_queue.cc:261] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } } peers { permanent_uuid: "13be8ace710348bca7c547205274be5e" member_type: NON_VOTER last_known_addr { host: "127.18.102.3" port: 45355 } attrs { promote: true } }
I20261116 08:58:48.800542 23397 ts_tablet_manager.cc:1480] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20261116 08:58:48.802430 23102 tablet_copy_service.cc:342] P c6acbb08cfd14f12bb2c18a08e498b78: Request end of tablet copy session 13be8ace710348bca7c547205274be5e-2bbeb51ea6804e9b9c9be801ff344029 received from {username='slave'} at 127.0.0.1:42212
I20261116 08:58:48.802891 23102 tablet_copy_service.cc:434] P c6acbb08cfd14f12bb2c18a08e498b78: ending tablet copy session 13be8ace710348bca7c547205274be5e-2bbeb51ea6804e9b9c9be801ff344029 on tablet 2bbeb51ea6804e9b9c9be801ff344029 with peer 13be8ace710348bca7c547205274be5e
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d45ca5c653ae4ed78f38ddb509c4aaea | 127.18.102.62:40211 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0eef0132476a4eb09475312095772337 | 127.18.102.1:35919 | HEALTHY | <none> | 0 | 0
13be8ace710348bca7c547205274be5e | 127.18.102.3:45355 | HEALTHY | <none> | 0 | 0
c5a51f9f9ba246f7961b8fc5e4734f92 | 127.18.102.4:40663 | HEALTHY | <none> | 0 | 0
c6acbb08cfd14f12bb2c18a08e498b78 | 127.18.102.2:41687 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2bbeb51ea6804e9b9c9be801ff344029 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
0eef0132476a4eb09475312095772337 (127.18.102.1:35919): RUNNING
c6acbb08cfd14f12bb2c18a08e498b78 (127.18.102.2:41687): RUNNING [LEADER]
13be8ace710348bca7c547205274be5e (127.18.102.3:45355): missing [NONVOTER]
All reported replicas are:
A = 0eef0132476a4eb09475312095772337
B = c6acbb08cfd14f12bb2c18a08e498b78
C = 13be8ace710348bca7c547205274be5e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C~ | 1 | 3 | Yes
B | A B* C~ | 1 | 3 | Yes
C | A B C~ | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | c5a51f9f9ba246f7961b8fc5e4734f92 | 127.18.102.4:40663 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20261116 08:58:49.134346 18840 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261116 08:58:49.195333 23166 raft_consensus.cc:1227] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20261116 08:58:49.599541 23082 tablet_service.cc:1456] Tablet server has 1 leaders and 0 scanners
I20261116 08:58:49.605432 23007 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:49.607438 23230 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:49.609205 23156 tablet_service.cc:1456] Tablet server has 0 leaders and 0 scanners
I20261116 08:58:49.625146 23339 raft_consensus.cc:1074] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78: attempting to promote NON_VOTER 13be8ace710348bca7c547205274be5e to VOTER
I20261116 08:58:49.626586 23339 consensus_queue.cc:238] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } } peers { permanent_uuid: "13be8ace710348bca7c547205274be5e" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 45355 } attrs { promote: false } }
I20261116 08:58:49.632050 23166 raft_consensus.cc:1285] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e [term 1 LEARNER]: Refusing update from remote peer c6acbb08cfd14f12bb2c18a08e498b78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20261116 08:58:49.633668 23016 raft_consensus.cc:1285] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 1 FOLLOWER]: Refusing update from remote peer c6acbb08cfd14f12bb2c18a08e498b78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20261116 08:58:49.633893 23412 consensus_queue.cc:1049] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13be8ace710348bca7c547205274be5e" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261116 08:58:49.636235 23412 consensus_queue.cc:1049] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261116 08:58:49.654970 23413 raft_consensus.cc:2983] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 13be8ace710348bca7c547205274be5e (127.18.102.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } } peers { permanent_uuid: "13be8ace710348bca7c547205274be5e" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 45355 } attrs { promote: false } } }
I20261116 08:58:49.664280 23017 raft_consensus.cc:2983] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 13be8ace710348bca7c547205274be5e (127.18.102.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } } peers { permanent_uuid: "13be8ace710348bca7c547205274be5e" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 45355 } attrs { promote: false } } }
I20261116 08:58:49.701459 22915 catalog_manager.cc:5840] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 reported cstate change: config changed from index 3 to 4, 13be8ace710348bca7c547205274be5e (127.18.102.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13be8ace710348bca7c547205274be5e" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 45355 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20261116 08:58:49.707237 23166 raft_consensus.cc:2983] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 13be8ace710348bca7c547205274be5e (127.18.102.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0eef0132476a4eb09475312095772337" member_type: VOTER last_known_addr { host: "127.18.102.1" port: 35919 } } peers { permanent_uuid: "c6acbb08cfd14f12bb2c18a08e498b78" member_type: VOTER last_known_addr { host: "127.18.102.2" port: 41687 } } peers { permanent_uuid: "13be8ace710348bca7c547205274be5e" member_type: VOTER last_known_addr { host: "127.18.102.3" port: 45355 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d45ca5c653ae4ed78f38ddb509c4aaea | 127.18.102.62:40211 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0eef0132476a4eb09475312095772337 | 127.18.102.1:35919 | HEALTHY | <none> | 0 | 0
13be8ace710348bca7c547205274be5e | 127.18.102.3:45355 | HEALTHY | <none> | 0 | 0
c5a51f9f9ba246f7961b8fc5e4734f92 | 127.18.102.4:40663 | HEALTHY | <none> | 0 | 0
c6acbb08cfd14f12bb2c18a08e498b78 | 127.18.102.2:41687 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | c5a51f9f9ba246f7961b8fc5e4734f92 | 127.18.102.4:40663 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20261116 08:58:50.043558 18840 log_verifier.cc:126] Checking tablet 2bbeb51ea6804e9b9c9be801ff344029
I20261116 08:58:50.125332 18840 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 2bbeb51ea6804e9b9c9be801ff344029
I20261116 08:58:50.129776 18840 tablet_server.cc:179] TabletServer@127.18.102.1:0 shutting down...
I20261116 08:58:50.175626 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:50.176371 18840 tablet_replica.cc:333] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337: stopping tablet replica
I20261116 08:58:50.177078 18840 raft_consensus.cc:2261] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:50.177950 18840 raft_consensus.cc:2290] T 2bbeb51ea6804e9b9c9be801ff344029 P 0eef0132476a4eb09475312095772337 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:50.200639 18840 tablet_server.cc:196] TabletServer@127.18.102.1:0 shutdown complete.
I20261116 08:58:50.216634 18840 tablet_server.cc:179] TabletServer@127.18.102.2:0 shutting down...
W20261116 08:58:50.249446 23055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.102.1:35919: connect: Connection refused (error 111) [suppressed 5 similar messages]
I20261116 08:58:50.252311 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:50.253372 18840 tablet_replica.cc:333] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78: stopping tablet replica
W20261116 08:58:50.253593 23055 consensus_peers.cc:597] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 -> Peer 0eef0132476a4eb09475312095772337 (127.18.102.1:35919): Couldn't send request to peer 0eef0132476a4eb09475312095772337. Status: Network error: Client connection negotiation failed: client connection to 127.18.102.1:35919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261116 08:58:50.254256 18840 raft_consensus.cc:2261] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:50.255698 18840 raft_consensus.cc:2290] T 2bbeb51ea6804e9b9c9be801ff344029 P c6acbb08cfd14f12bb2c18a08e498b78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:50.278903 18840 tablet_server.cc:196] TabletServer@127.18.102.2:0 shutdown complete.
I20261116 08:58:50.292248 18840 tablet_server.cc:179] TabletServer@127.18.102.3:0 shutting down...
I20261116 08:58:50.319558 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261116 08:58:50.320375 18840 tablet_replica.cc:333] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e: stopping tablet replica
I20261116 08:58:50.320953 18840 raft_consensus.cc:2261] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20261116 08:58:50.321592 18840 raft_consensus.cc:2290] T 2bbeb51ea6804e9b9c9be801ff344029 P 13be8ace710348bca7c547205274be5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:50.341920 18840 tablet_server.cc:196] TabletServer@127.18.102.3:0 shutdown complete.
I20261116 08:58:50.354614 18840 tablet_server.cc:179] TabletServer@127.18.102.4:0 shutting down...
I20261116 08:58:50.392678 18840 ts_tablet_manager.cc:1553] Shutting down tablet manager...
W20261116 08:58:51.409427 18840 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 23261)
I20261116 08:58:52.249612 18840 tablet_server.cc:196] TabletServer@127.18.102.4:0 shutdown complete.
I20261116 08:58:52.260578 18840 master.cc:585] Master@127.18.102.62:40211 shutting down...
I20261116 08:58:52.278875 18840 raft_consensus.cc:2261] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [term 1 LEADER]: Raft consensus shutting down.
I20261116 08:58:52.279448 18840 raft_consensus.cc:2290] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 08:58:52.279759 18840 tablet_replica.cc:333] T 00000000000000000000000000000000 P d45ca5c653ae4ed78f38ddb509c4aaea: stopping tablet replica
I20261116 08:58:52.297263 18840 master.cc:607] Master@127.18.102.62:40211 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (6495 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (9643 ms total)
[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (62766 ms total)
[ PASSED ] 11 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[ FAILED ] 1 test, listed below:
[ FAILED ] TsTabletManagerITest.TestTableStats
1 FAILED TEST
I20261116 08:58:52.336249 18840 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~10 seconds ago
I20261116 08:58:52.336597 18840 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:648: suppressed but not reported on 74 messages since previous log ~57 seconds ago
I20261116 08:58:52.336901 18840 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2441: suppressed but not reported on 6 messages since previous log ~13 seconds ago