[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20261120 03:02:50.040824 385 thread.cc:642] rpc reactor (reactor) Time spent creating pthread: real 1.117s user 0.418s sys 0.699s
W20261120 03:02:50.041303 385 thread.cc:609] rpc reactor (reactor) Time spent starting thread: real 1.118s user 0.418s sys 0.699s
I20261120 03:02:50.046075 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:41885
I20261120 03:02:50.047827 385 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 03:02:50.049088 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:50.062438 396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:50.065327 397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:50.152732 385 server_base.cc:1061] running on GCE node
W20261120 03:02:50.152817 399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:50.154024 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:50.154284 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:50.154474 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143770154451 us; error 0 us; skew 500 ppm
I20261120 03:02:50.155157 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:50.159634 385 webserver.cc:577] Webserver started at http://127.0.96.126:43295/ using document root <none> and password file <none>
I20261120 03:02:50.160435 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:50.160648 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:50.161087 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:50.170782 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "e21e1011c1784b58ad643adc3e231996"
format_stamp: "Formatted at 2026-11-20 03:02:50 on dist-test-slave-54pf"
I20261120 03:02:50.177853 385 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20261120 03:02:50.182910 404 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:50.183894 385 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20261120 03:02:50.184273 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "e21e1011c1784b58ad643adc3e231996"
format_stamp: "Formatted at 2026-11-20 03:02:50 on dist-test-slave-54pf"
I20261120 03:02:50.184577 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:50.205601 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:50.206974 385 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 03:02:50.207396 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:50.280997 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:41885
I20261120 03:02:50.281056 455 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:41885 every 8 connection(s)
I20261120 03:02:50.287717 456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:50.309675 456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996: Bootstrap starting.
I20261120 03:02:50.316427 456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:50.318010 456 log.cc:826] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996: Log is configured to *not* fsync() on all Append() calls
I20261120 03:02:50.322595 456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996: No bootstrap required, opened a new log
I20261120 03:02:50.342831 456 raft_consensus.cc:366] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21e1011c1784b58ad643adc3e231996" member_type: VOTER }
I20261120 03:02:50.343411 456 raft_consensus.cc:392] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:50.343658 456 raft_consensus.cc:787] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e21e1011c1784b58ad643adc3e231996, State: Initialized, Role: FOLLOWER
I20261120 03:02:50.344659 456 consensus_queue.cc:261] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21e1011c1784b58ad643adc3e231996" member_type: VOTER }
I20261120 03:02:50.345237 456 raft_consensus.cc:407] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:02:50.345546 456 raft_consensus.cc:534] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:02:50.345932 456 raft_consensus.cc:3296] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:50.348452 456 raft_consensus.cc:556] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21e1011c1784b58ad643adc3e231996" member_type: VOTER }
I20261120 03:02:50.352891 456 leader_election.cc:304] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e21e1011c1784b58ad643adc3e231996; no voters:
I20261120 03:02:50.353484 456 leader_election.cc:290] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:02:50.353668 459 raft_consensus.cc:3024] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:02:50.356720 459 raft_consensus.cc:744] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 1 LEADER]: Becoming Leader. State: Replica: e21e1011c1784b58ad643adc3e231996, State: Running, Role: LEADER
I20261120 03:02:50.357458 459 consensus_queue.cc:238] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21e1011c1784b58ad643adc3e231996" member_type: VOTER }
I20261120 03:02:50.358515 456 sys_catalog.cc:565] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:02:50.367194 460 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e21e1011c1784b58ad643adc3e231996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21e1011c1784b58ad643adc3e231996" member_type: VOTER } }
I20261120 03:02:50.369271 460 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:50.367074 461 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e21e1011c1784b58ad643adc3e231996. Latest consensus state: current_term: 1 leader_uuid: "e21e1011c1784b58ad643adc3e231996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21e1011c1784b58ad643adc3e231996" member_type: VOTER } }
I20261120 03:02:50.370153 461 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:50.373601 468 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:02:50.385291 468 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:02:50.387223 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:02:50.400343 468 catalog_manager.cc:1398] Generated new cluster ID: 126ebf1cbedf4513b5180b763d1ced4a
I20261120 03:02:50.400591 468 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:02:50.424769 468 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:02:50.426559 468 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:02:50.449887 468 catalog_manager.cc:6237] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996: Generated new TSK 0
I20261120 03:02:50.450917 468 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:02:50.517576 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:50.527235 478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:50.529088 479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:50.533397 385 server_base.cc:1061] running on GCE node
W20261120 03:02:50.533456 481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:50.534327 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:50.534543 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:50.534742 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143770534725 us; error 0 us; skew 500 ppm
I20261120 03:02:50.535307 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:50.539877 385 webserver.cc:577] Webserver started at http://127.0.96.65:39301/ using document root <none> and password file <none>
I20261120 03:02:50.540427 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:50.540663 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:50.540948 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:50.542151 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "5447aa6248eb48c29da5c94ead234e39"
format_stamp: "Formatted at 2026-11-20 03:02:50 on dist-test-slave-54pf"
I20261120 03:02:50.547582 385 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20261120 03:02:50.552083 486 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:50.553009 385 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20261120 03:02:50.553297 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "5447aa6248eb48c29da5c94ead234e39"
format_stamp: "Formatted at 2026-11-20 03:02:50 on dist-test-slave-54pf"
I20261120 03:02:50.553609 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:50.577068 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:50.578270 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:50.581223 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:50.585417 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:50.585598 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:50.585877 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:50.586064 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:50.696205 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:38505
I20261120 03:02:50.696272 548 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:38505 every 8 connection(s)
I20261120 03:02:50.703512 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:50.715857 553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:50.725775 554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:50.728582 385 server_base.cc:1061] running on GCE node
W20261120 03:02:50.728572 556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:50.729579 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:50.729799 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:50.730012 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143770729977 us; error 0 us; skew 500 ppm
I20261120 03:02:50.730557 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:50.733397 385 webserver.cc:577] Webserver started at http://127.0.96.66:38963/ using document root <none> and password file <none>
I20261120 03:02:50.733947 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:50.734176 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:50.734490 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:50.735833 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "ecfa61e8132642ac916dded8a129376a"
format_stamp: "Formatted at 2026-11-20 03:02:50 on dist-test-slave-54pf"
I20261120 03:02:50.737684 549 heartbeater.cc:345] Connected to a master server at 127.0.96.126:41885
I20261120 03:02:50.738142 549 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:50.739223 549 heartbeater.cc:508] Master 127.0.96.126:41885 requested a full tablet report, sending...
I20261120 03:02:50.742740 385 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20261120 03:02:50.743119 421 ts_manager.cc:210] Registered new tserver with Master: 5447aa6248eb48c29da5c94ead234e39 (127.0.96.65:38505)
I20261120 03:02:50.746047 421 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58360
I20261120 03:02:50.747254 561 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:50.748175 385 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20261120 03:02:50.748461 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "ecfa61e8132642ac916dded8a129376a"
format_stamp: "Formatted at 2026-11-20 03:02:50 on dist-test-slave-54pf"
I20261120 03:02:50.748803 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:50.767460 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:50.768630 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:50.770954 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:50.774176 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:50.774343 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:50.774616 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:50.774792 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:50.850265 549 heartbeater.cc:500] Master 127.0.96.126:41885 was elected leader, sending a full tablet report...
I20261120 03:02:50.879045 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:35013
I20261120 03:02:50.879148 623 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:35013 every 8 connection(s)
I20261120 03:02:50.894590 624 heartbeater.cc:345] Connected to a master server at 127.0.96.126:41885
I20261120 03:02:50.894910 624 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:50.895560 624 heartbeater.cc:508] Master 127.0.96.126:41885 requested a full tablet report, sending...
I20261120 03:02:50.897472 421 ts_manager.cc:210] Registered new tserver with Master: ecfa61e8132642ac916dded8a129376a (127.0.96.66:35013)
I20261120 03:02:50.898782 385 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.015125361s
I20261120 03:02:50.899519 421 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58364
I20261120 03:02:50.927897 421 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
range_schema {
columns {
name: "key"
}
}
}
W20261120 03:02:50.945463 421 catalog_manager.cc:7244] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20261120 03:02:50.997280 589 tablet_service.cc:1495] Processing CreateTablet for tablet b8b8567da20c4ae1a2a6c155b887254b (DEFAULT_TABLE table=test-table [id=3790e4dc82734b1d9ff1d740a1467d50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261120 03:02:50.999182 588 tablet_service.cc:1495] Processing CreateTablet for tablet 62c70f83df9048fe93523ea6df2addc9 (DEFAULT_TABLE table=test-table [id=3790e4dc82734b1d9ff1d740a1467d50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261120 03:02:50.999801 589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b8567da20c4ae1a2a6c155b887254b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:51.000559 588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62c70f83df9048fe93523ea6df2addc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:51.000428 514 tablet_service.cc:1495] Processing CreateTablet for tablet b8b8567da20c4ae1a2a6c155b887254b (DEFAULT_TABLE table=test-table [id=3790e4dc82734b1d9ff1d740a1467d50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261120 03:02:51.001675 514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b8567da20c4ae1a2a6c155b887254b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:51.002096 624 heartbeater.cc:500] Master 127.0.96.126:41885 was elected leader, sending a full tablet report...
I20261120 03:02:51.003746 513 tablet_service.cc:1495] Processing CreateTablet for tablet 62c70f83df9048fe93523ea6df2addc9 (DEFAULT_TABLE table=test-table [id=3790e4dc82734b1d9ff1d740a1467d50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261120 03:02:51.004851 513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62c70f83df9048fe93523ea6df2addc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:51.034216 640 tablet_bootstrap.cc:492] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a: Bootstrap starting.
I20261120 03:02:51.041764 641 tablet_bootstrap.cc:492] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39: Bootstrap starting.
I20261120 03:02:51.042625 640 tablet_bootstrap.cc:654] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:51.048375 640 tablet_bootstrap.cc:492] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a: No bootstrap required, opened a new log
I20261120 03:02:51.048725 640 ts_tablet_manager.cc:1449] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20261120 03:02:51.049633 641 tablet_bootstrap.cc:654] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:51.051049 640 raft_consensus.cc:366] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.051544 640 raft_consensus.cc:787] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Initialized, Role: FOLLOWER
I20261120 03:02:51.052090 640 consensus_queue.cc:261] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.054401 640 ts_tablet_manager.cc:1480] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20261120 03:02:51.055697 640 tablet_bootstrap.cc:492] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: Bootstrap starting.
I20261120 03:02:51.059751 641 tablet_bootstrap.cc:492] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39: No bootstrap required, opened a new log
I20261120 03:02:51.060336 641 ts_tablet_manager.cc:1449] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20261120 03:02:51.062566 640 tablet_bootstrap.cc:654] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:51.063140 641 raft_consensus.cc:366] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
I20261120 03:02:51.063854 641 raft_consensus.cc:787] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5447aa6248eb48c29da5c94ead234e39, State: Initialized, Role: FOLLOWER
I20261120 03:02:51.064563 641 consensus_queue.cc:261] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
I20261120 03:02:51.082293 641 ts_tablet_manager.cc:1480] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39: Time spent starting tablet: real 0.021s user 0.008s sys 0.008s
I20261120 03:02:51.083113 641 tablet_bootstrap.cc:492] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39: Bootstrap starting.
I20261120 03:02:51.084401 640 tablet_bootstrap.cc:492] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: No bootstrap required, opened a new log
I20261120 03:02:51.084937 640 ts_tablet_manager.cc:1449] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.010s
I20261120 03:02:51.087936 640 raft_consensus.cc:366] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
I20261120 03:02:51.088629 640 raft_consensus.cc:787] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Initialized, Role: FOLLOWER
I20261120 03:02:51.089255 640 consensus_queue.cc:261] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
I20261120 03:02:51.090127 641 tablet_bootstrap.cc:654] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:51.092713 640 ts_tablet_manager.cc:1480] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20261120 03:02:51.105151 641 tablet_bootstrap.cc:492] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39: No bootstrap required, opened a new log
I20261120 03:02:51.105708 641 ts_tablet_manager.cc:1449] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.005s
I20261120 03:02:51.108824 641 raft_consensus.cc:366] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.109308 641 raft_consensus.cc:787] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5447aa6248eb48c29da5c94ead234e39, State: Initialized, Role: FOLLOWER
I20261120 03:02:51.109731 641 consensus_queue.cc:261] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.111485 641 ts_tablet_manager.cc:1480] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20261120 03:02:51.126739 385 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20261120 03:02:51.127024 385 raft_consensus.cc:433] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Emulating election...
I20261120 03:02:51.127271 385 raft_consensus.cc:3296] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:51.129714 385 raft_consensus.cc:744] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Becoming Leader. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:51.130369 385 consensus_queue.cc:238] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
W20261120 03:02:51.136704 625 tablet.cc:2554] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261120 03:02:51.138794 385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:51.145825 421 catalog_manager.cc:5856] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a reported cstate change: term changed from 0 to 1, leader changed from <none> to ecfa61e8132642ac916dded8a129376a (127.0.96.66). New cstate: current_term: 1 leader_uuid: "ecfa61e8132642ac916dded8a129376a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } health_report { overall_health: HEALTHY } } }
I20261120 03:02:51.155512 524 raft_consensus.cc:3296] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:51.164139 643 consensus_queue.cc:1049] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261120 03:02:51.171869 385 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20261120 03:02:51.276032 385 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20261120 03:02:51.276361 385 raft_consensus.cc:433] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Emulating election...
I20261120 03:02:51.276646 385 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:51.277360 385 raft_consensus.cc:744] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Becoming Leader. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:51.278007 385 consensus_queue.cc:238] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.282943 385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:51.284026 421 catalog_manager.cc:5856] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a reported cstate change: term changed from 0 to 1, leader changed from <none> to ecfa61e8132642ac916dded8a129376a (127.0.96.66). New cstate: current_term: 1 leader_uuid: "ecfa61e8132642ac916dded8a129376a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20261120 03:02:51.285388 523 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:51.295140 643 consensus_queue.cc:1049] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:02:51.297107 385 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20261120 03:02:51.601078 385 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20261120 03:02:51.601366 385 raft_consensus.cc:433] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Emulating election...
I20261120 03:02:51.601702 385 raft_consensus.cc:3291] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:02:51.602078 385 raft_consensus.cc:787] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:51.602891 385 consensus_queue.cc:261] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.603915 385 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:02:51.604655 385 raft_consensus.cc:744] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 2 LEADER]: Becoming Leader. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:51.605245 385 consensus_queue.cc:238] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.609562 385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:51.610901 420 catalog_manager.cc:5856] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "ecfa61e8132642ac916dded8a129376a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20261120 03:02:51.611577 523 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:02:51.612534 523 raft_consensus.cc:1338] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 2 FOLLOWER]: Refusing update from remote peer ecfa61e8132642ac916dded8a129376a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261120 03:02:51.614773 385 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20261120 03:02:51.619625 650 consensus_queue.cc:1049] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 03:02:51.718739 385 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20261120 03:02:51.719038 385 raft_consensus.cc:433] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 2 FOLLOWER]: Emulating election...
I20261120 03:02:51.719286 385 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:02:51.721539 385 raft_consensus.cc:744] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 3 LEADER]: Becoming Leader. State: Replica: 5447aa6248eb48c29da5c94ead234e39, State: Running, Role: LEADER
I20261120 03:02:51.722182 385 consensus_queue.cc:238] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.726184 385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:51.727814 420 catalog_manager.cc:5856] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 reported cstate change: term changed from 2 to 3, leader changed from ecfa61e8132642ac916dded8a129376a (127.0.96.66) to 5447aa6248eb48c29da5c94ead234e39 (127.0.96.65). New cstate: current_term: 3 leader_uuid: "5447aa6248eb48c29da5c94ead234e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } health_report { overall_health: HEALTHY } } }
I20261120 03:02:51.732304 385 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20261120 03:02:51.752517 599 raft_consensus.cc:3291] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 2 LEADER]: Stepping down as leader of term 2
I20261120 03:02:51.752867 599 raft_consensus.cc:787] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:51.753482 599 consensus_queue.cc:261] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.754279 599 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:02:51.755033 599 raft_consensus.cc:1338] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 3 FOLLOWER]: Refusing update from remote peer 5447aa6248eb48c29da5c94ead234e39: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20261120 03:02:51.759758 646 consensus_queue.cc:1049] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:02:52.036280 385 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20261120 03:02:52.036523 385 raft_consensus.cc:433] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 1 FOLLOWER]: Emulating election...
I20261120 03:02:52.036758 385 raft_consensus.cc:3296] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:02:52.037344 385 raft_consensus.cc:744] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 2 LEADER]: Becoming Leader. State: Replica: 5447aa6248eb48c29da5c94ead234e39, State: Running, Role: LEADER
I20261120 03:02:52.037969 385 consensus_queue.cc:238] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
I20261120 03:02:52.042675 419 catalog_manager.cc:5856] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 reported cstate change: term changed from 1 to 2, leader changed from ecfa61e8132642ac916dded8a129376a (127.0.96.66) to 5447aa6248eb48c29da5c94ead234e39 (127.0.96.65). New cstate: current_term: 2 leader_uuid: "5447aa6248eb48c29da5c94ead234e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } health_report { overall_health: UNKNOWN } } }
I20261120 03:02:52.044062 385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:52.046386 599 raft_consensus.cc:3291] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:02:52.046701 599 raft_consensus.cc:787] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:52.047257 599 consensus_queue.cc:261] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
I20261120 03:02:52.048110 599 raft_consensus.cc:3296] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:02:52.052510 599 raft_consensus.cc:1338] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 2 FOLLOWER]: Refusing update from remote peer 5447aa6248eb48c29da5c94ead234e39: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261120 03:02:52.055073 385 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20261120 03:02:52.055882 659 consensus_queue.cc:1049] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 03:02:52.159601 385 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20261120 03:02:52.159861 385 raft_consensus.cc:433] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 3 FOLLOWER]: Emulating election...
I20261120 03:02:52.160163 385 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:02:52.161406 385 raft_consensus.cc:744] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 4 LEADER]: Becoming Leader. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:52.162057 385 consensus_queue.cc:238] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:52.167339 385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:52.166570 420 catalog_manager.cc:5856] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a reported cstate change: term changed from 3 to 4, leader changed from 5447aa6248eb48c29da5c94ead234e39 (127.0.96.65) to ecfa61e8132642ac916dded8a129376a (127.0.96.66). New cstate: current_term: 4 leader_uuid: "ecfa61e8132642ac916dded8a129376a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20261120 03:02:52.169602 524 raft_consensus.cc:3291] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 3 LEADER]: Stepping down as leader of term 3
I20261120 03:02:52.169952 524 raft_consensus.cc:787] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5447aa6248eb48c29da5c94ead234e39, State: Running, Role: LEADER
I20261120 03:02:52.170550 524 consensus_queue.cc:261] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:52.171424 524 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:02:52.172834 524 raft_consensus.cc:1338] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 4 FOLLOWER]: Refusing update from remote peer ecfa61e8132642ac916dded8a129376a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20261120 03:02:52.185900 643 consensus_queue.cc:1049] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:02:52.188866 385 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20261120 03:02:52.494252 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:02:52.514456 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:02:52.515002 385 tablet_replica.cc:333] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39: stopping tablet replica
I20261120 03:02:52.515501 385 raft_consensus.cc:2440] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:02:52.515935 385 raft_consensus.cc:2469] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:52.517999 385 tablet_replica.cc:333] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39: stopping tablet replica
I20261120 03:02:52.518399 385 raft_consensus.cc:2440] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 2 LEADER]: Raft consensus shutting down.
I20261120 03:02:52.518859 385 raft_consensus.cc:2469] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:52.595891 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:02:52.604504 385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:02:52.620702 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:02:52.621222 385 tablet_replica.cc:333] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: stopping tablet replica
I20261120 03:02:52.621672 385 raft_consensus.cc:2440] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 2 FOLLOWER]: Raft consensus shutting down.
I20261120 03:02:52.622051 385 raft_consensus.cc:2469] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:52.623639 385 tablet_replica.cc:333] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a: stopping tablet replica
I20261120 03:02:52.624065 385 raft_consensus.cc:2440] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 4 LEADER]: Raft consensus shutting down.
I20261120 03:02:52.624585 385 raft_consensus.cc:2469] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:52.632028 385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:02:52.639272 385 master.cc:585] Master@127.0.96.126:41885 shutting down...
I20261120 03:02:52.651302 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:02:52.651695 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:52.652007 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996: stopping tablet replica
I20261120 03:02:52.670928 385 master.cc:607] Master@127.0.96.126:41885 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (3817 ms)
[ RUN ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20261120 03:02:52.703197 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:41183
I20261120 03:02:52.704322 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:52.708534 669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:52.709473 670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:52.712595 385 server_base.cc:1061] running on GCE node
W20261120 03:02:52.713920 672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:52.714552 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:52.714717 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:52.714823 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143772714812 us; error 0 us; skew 500 ppm
I20261120 03:02:52.715188 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:52.717239 385 webserver.cc:577] Webserver started at http://127.0.96.126:33509/ using document root <none> and password file <none>
I20261120 03:02:52.717592 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:52.717722 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:52.717895 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:52.718743 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "d085122c19f84ee8b64d592e736fe55e"
format_stamp: "Formatted at 2026-11-20 03:02:52 on dist-test-slave-54pf"
I20261120 03:02:52.722698 385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261120 03:02:52.725286 677 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:52.725940 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 03:02:52.726212 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "d085122c19f84ee8b64d592e736fe55e"
format_stamp: "Formatted at 2026-11-20 03:02:52 on dist-test-slave-54pf"
I20261120 03:02:52.726452 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:52.738395 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:52.739239 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:52.770321 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:41183
I20261120 03:02:52.770411 728 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:41183 every 8 connection(s)
I20261120 03:02:52.773727 729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:52.782223 729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e: Bootstrap starting.
I20261120 03:02:52.785889 729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:52.789340 729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e: No bootstrap required, opened a new log
I20261120 03:02:52.790975 729 raft_consensus.cc:366] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085122c19f84ee8b64d592e736fe55e" member_type: VOTER }
I20261120 03:02:52.791342 729 raft_consensus.cc:392] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:52.791579 729 raft_consensus.cc:787] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d085122c19f84ee8b64d592e736fe55e, State: Initialized, Role: FOLLOWER
I20261120 03:02:52.792095 729 consensus_queue.cc:261] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085122c19f84ee8b64d592e736fe55e" member_type: VOTER }
I20261120 03:02:52.792557 729 raft_consensus.cc:407] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:02:52.792801 729 raft_consensus.cc:534] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:02:52.793094 729 raft_consensus.cc:3296] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:52.794075 729 raft_consensus.cc:556] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085122c19f84ee8b64d592e736fe55e" member_type: VOTER }
I20261120 03:02:52.797783 729 leader_election.cc:304] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d085122c19f84ee8b64d592e736fe55e; no voters:
I20261120 03:02:52.798237 729 leader_election.cc:290] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:02:52.798336 732 raft_consensus.cc:3024] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:02:52.799775 732 raft_consensus.cc:744] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 1 LEADER]: Becoming Leader. State: Replica: d085122c19f84ee8b64d592e736fe55e, State: Running, Role: LEADER
I20261120 03:02:52.800534 732 consensus_queue.cc:238] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085122c19f84ee8b64d592e736fe55e" member_type: VOTER }
I20261120 03:02:52.801303 729 sys_catalog.cc:565] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:02:52.803035 733 sys_catalog.cc:455] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d085122c19f84ee8b64d592e736fe55e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085122c19f84ee8b64d592e736fe55e" member_type: VOTER } }
I20261120 03:02:52.803068 734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d085122c19f84ee8b64d592e736fe55e. Latest consensus state: current_term: 1 leader_uuid: "d085122c19f84ee8b64d592e736fe55e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085122c19f84ee8b64d592e736fe55e" member_type: VOTER } }
I20261120 03:02:52.803674 734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:52.803710 733 sys_catalog.cc:458] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:52.806228 737 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:02:52.810657 737 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:02:52.815560 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:02:52.818178 737 catalog_manager.cc:1398] Generated new cluster ID: 3c40363c2d3043b081be9496e3d3f8af
I20261120 03:02:52.818419 737 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:02:52.833510 737 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:02:52.834632 737 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:02:52.848431 737 catalog_manager.cc:6237] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e: Generated new TSK 0
I20261120 03:02:52.848946 737 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:02:52.881249 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:52.885972 750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:52.887203 751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:52.888425 753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:52.889441 385 server_base.cc:1061] running on GCE node
I20261120 03:02:52.890195 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:52.890373 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:52.890478 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143772890468 us; error 0 us; skew 500 ppm
I20261120 03:02:52.890854 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:52.892714 385 webserver.cc:577] Webserver started at http://127.0.96.65:36055/ using document root <none> and password file <none>
I20261120 03:02:52.893085 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:52.893235 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:52.893430 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:52.894299 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "0841085fdf594225b62f17e075964efc"
format_stamp: "Formatted at 2026-11-20 03:02:52 on dist-test-slave-54pf"
I20261120 03:02:52.897943 385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261120 03:02:52.900786 758 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:52.901484 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 03:02:52.901763 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "0841085fdf594225b62f17e075964efc"
format_stamp: "Formatted at 2026-11-20 03:02:52 on dist-test-slave-54pf"
I20261120 03:02:52.902033 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:52.921432 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:52.922277 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:52.923404 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:52.925359 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:52.925571 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:52.925797 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:52.925963 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:52.958354 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:35779
I20261120 03:02:52.958458 820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:35779 every 8 connection(s)
I20261120 03:02:52.962819 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:52.969135 825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:52.969609 826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:52.973672 828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:52.974501 385 server_base.cc:1061] running on GCE node
I20261120 03:02:52.975126 821 heartbeater.cc:345] Connected to a master server at 127.0.96.126:41183
I20261120 03:02:52.975299 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20261120 03:02:52.975476 821 heartbeater.cc:462] Registering TS with master...
W20261120 03:02:52.975586 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:52.975855 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143772975838 us; error 0 us; skew 500 ppm
I20261120 03:02:52.976331 821 heartbeater.cc:508] Master 127.0.96.126:41183 requested a full tablet report, sending...
I20261120 03:02:52.976385 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:52.978284 694 ts_manager.cc:210] Registered new tserver with Master: 0841085fdf594225b62f17e075964efc (127.0.96.65:35779)
I20261120 03:02:52.978859 385 webserver.cc:577] Webserver started at http://127.0.96.66:33631/ using document root <none> and password file <none>
I20261120 03:02:52.979329 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:52.979532 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:52.979789 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:52.980060 694 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51876
I20261120 03:02:52.980968 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "afdf2aa455bd4fcdbd94920f8c83c062"
format_stamp: "Formatted at 2026-11-20 03:02:52 on dist-test-slave-54pf"
I20261120 03:02:52.984881 385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20261120 03:02:52.987715 833 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:52.988490 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20261120 03:02:52.988757 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "afdf2aa455bd4fcdbd94920f8c83c062"
format_stamp: "Formatted at 2026-11-20 03:02:52 on dist-test-slave-54pf"
I20261120 03:02:52.989024 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:53.006491 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:53.007314 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:53.008454 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:53.010470 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:53.010668 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:53.010877 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:53.011027 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:53.044147 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:41527
I20261120 03:02:53.044269 895 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:41527 every 8 connection(s)
I20261120 03:02:53.056586 896 heartbeater.cc:345] Connected to a master server at 127.0.96.126:41183
I20261120 03:02:53.056922 896 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:53.057722 896 heartbeater.cc:508] Master 127.0.96.126:41183 requested a full tablet report, sending...
I20261120 03:02:53.059435 694 ts_manager.cc:210] Registered new tserver with Master: afdf2aa455bd4fcdbd94920f8c83c062 (127.0.96.66:41527)
I20261120 03:02:53.060256 385 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.013017822s
I20261120 03:02:53.060889 694 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51878
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:422: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20261120 03:02:53.079242 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:02:53.082311 821 heartbeater.cc:500] Master 127.0.96.126:41183 was elected leader, sending a full tablet report...
I20261120 03:02:53.094622 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:02:53.109112 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:02:53.115116 385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:02:53.127990 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:02:53.142519 385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:02:53.148401 385 master.cc:585] Master@127.0.96.126:41183 shutting down...
I20261120 03:02:53.218262 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:02:53.218716 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:53.219066 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e: stopping tablet replica
I20261120 03:02:53.227041 385 master.cc:607] Master@127.0.96.126:41183 shutdown complete.
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (546 ms)
[ RUN ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20261120 03:02:53.247874 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:43817
I20261120 03:02:53.248777 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:53.252965 909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:53.254076 910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:53.255172 912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:53.255730 385 server_base.cc:1061] running on GCE node
I20261120 03:02:53.256554 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:53.256721 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:53.256832 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143773256822 us; error 0 us; skew 500 ppm
I20261120 03:02:53.257277 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:53.259178 385 webserver.cc:577] Webserver started at http://127.0.96.126:46515/ using document root <none> and password file <none>
I20261120 03:02:53.259598 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:53.259807 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:53.259994 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:53.261003 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "88e7003c263548a1a47d5399f68474db"
format_stamp: "Formatted at 2026-11-20 03:02:53 on dist-test-slave-54pf"
I20261120 03:02:53.264917 385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20261120 03:02:53.267578 917 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:53.268329 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261120 03:02:53.268589 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "88e7003c263548a1a47d5399f68474db"
format_stamp: "Formatted at 2026-11-20 03:02:53 on dist-test-slave-54pf"
I20261120 03:02:53.268865 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:53.293268 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:53.294150 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:53.325850 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:43817
I20261120 03:02:53.325927 968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:43817 every 8 connection(s)
I20261120 03:02:53.329370 969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:53.337953 969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db: Bootstrap starting.
I20261120 03:02:53.341802 969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:53.344969 969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db: No bootstrap required, opened a new log
I20261120 03:02:53.346736 969 raft_consensus.cc:366] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e7003c263548a1a47d5399f68474db" member_type: VOTER }
I20261120 03:02:53.347084 969 raft_consensus.cc:392] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:53.347312 969 raft_consensus.cc:787] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e7003c263548a1a47d5399f68474db, State: Initialized, Role: FOLLOWER
I20261120 03:02:53.347862 969 consensus_queue.cc:261] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e7003c263548a1a47d5399f68474db" member_type: VOTER }
I20261120 03:02:53.348318 969 raft_consensus.cc:407] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:02:53.348536 969 raft_consensus.cc:534] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:02:53.348821 969 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:53.349818 969 raft_consensus.cc:556] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e7003c263548a1a47d5399f68474db" member_type: VOTER }
I20261120 03:02:53.353677 969 leader_election.cc:304] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e7003c263548a1a47d5399f68474db; no voters:
I20261120 03:02:53.354095 969 leader_election.cc:290] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:02:53.354226 972 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:02:53.356175 972 raft_consensus.cc:744] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 1 LEADER]: Becoming Leader. State: Replica: 88e7003c263548a1a47d5399f68474db, State: Running, Role: LEADER
I20261120 03:02:53.356801 972 consensus_queue.cc:238] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e7003c263548a1a47d5399f68474db" member_type: VOTER }
I20261120 03:02:53.357623 969 sys_catalog.cc:565] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:02:53.359772 974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e7003c263548a1a47d5399f68474db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e7003c263548a1a47d5399f68474db" member_type: VOTER } }
I20261120 03:02:53.359845 973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e7003c263548a1a47d5399f68474db. Latest consensus state: current_term: 1 leader_uuid: "88e7003c263548a1a47d5399f68474db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e7003c263548a1a47d5399f68474db" member_type: VOTER } }
I20261120 03:02:53.360507 973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:53.360508 974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:53.363005 977 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:02:53.367304 977 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:02:53.373494 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:02:53.375591 977 catalog_manager.cc:1398] Generated new cluster ID: af685986090843a89a7f9861a371444d
I20261120 03:02:53.375824 977 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:02:53.398687 977 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:02:53.399760 977 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:02:53.413237 977 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db: Generated new TSK 0
I20261120 03:02:53.413787 977 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:02:53.439743 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:53.444774 990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:53.445981 991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:53.447175 993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:53.447427 385 server_base.cc:1061] running on GCE node
I20261120 03:02:53.448102 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:53.448333 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:53.448442 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143773448432 us; error 0 us; skew 500 ppm
I20261120 03:02:53.448880 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:53.450809 385 webserver.cc:577] Webserver started at http://127.0.96.65:45485/ using document root <none> and password file <none>
I20261120 03:02:53.451195 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:53.451337 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:53.451522 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:53.452531 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "a4e1c705ec064e7eb74a40c9a99fd34c"
format_stamp: "Formatted at 2026-11-20 03:02:53 on dist-test-slave-54pf"
I20261120 03:02:53.456312 385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20261120 03:02:53.459008 998 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:53.459815 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 03:02:53.460036 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "a4e1c705ec064e7eb74a40c9a99fd34c"
format_stamp: "Formatted at 2026-11-20 03:02:53 on dist-test-slave-54pf"
I20261120 03:02:53.460351 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:53.489024 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:53.489845 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:53.491003 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:53.492938 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:53.493147 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:53.493391 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:53.493544 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:53.527350 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:36709
I20261120 03:02:53.527491 1060 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:36709 every 8 connection(s)
I20261120 03:02:53.531857 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:53.538072 1065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:53.538353 1066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:53.542657 1068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:53.543097 385 server_base.cc:1061] running on GCE node
I20261120 03:02:53.543116 1061 heartbeater.cc:345] Connected to a master server at 127.0.96.126:43817
I20261120 03:02:53.543638 1061 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:53.544013 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:53.544279 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:53.544404 1061 heartbeater.cc:508] Master 127.0.96.126:43817 requested a full tablet report, sending...
I20261120 03:02:53.544466 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143773544447 us; error 0 us; skew 500 ppm
I20261120 03:02:53.545184 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:53.546520 934 ts_manager.cc:210] Registered new tserver with Master: a4e1c705ec064e7eb74a40c9a99fd34c (127.0.96.65:36709)
I20261120 03:02:53.547653 385 webserver.cc:577] Webserver started at http://127.0.96.66:36101/ using document root <none> and password file <none>
I20261120 03:02:53.548156 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:53.548355 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:53.548326 934 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36312
I20261120 03:02:53.548715 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:53.549844 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "d2f8f09810044404b290ce30d0f28d4b"
format_stamp: "Formatted at 2026-11-20 03:02:53 on dist-test-slave-54pf"
I20261120 03:02:53.553897 385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261120 03:02:53.556492 1073 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:53.557238 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20261120 03:02:53.557489 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "d2f8f09810044404b290ce30d0f28d4b"
format_stamp: "Formatted at 2026-11-20 03:02:53 on dist-test-slave-54pf"
I20261120 03:02:53.557770 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:53.575754 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:53.576598 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:53.577756 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:53.579698 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:53.579913 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:53.580224 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:53.580415 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:53.613437 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:46787
I20261120 03:02:53.613518 1135 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:46787 every 8 connection(s)
I20261120 03:02:53.625835 1136 heartbeater.cc:345] Connected to a master server at 127.0.96.126:43817
I20261120 03:02:53.626174 1136 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:53.626785 1136 heartbeater.cc:508] Master 127.0.96.126:43817 requested a full tablet report, sending...
I20261120 03:02:53.628545 934 ts_manager.cc:210] Registered new tserver with Master: d2f8f09810044404b290ce30d0f28d4b (127.0.96.66:46787)
I20261120 03:02:53.629223 385 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.013028441s
I20261120 03:02:53.629799 934 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36314
I20261120 03:02:53.648029 934 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20261120 03:02:53.649295 934 catalog_manager.cc:7244] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20261120 03:02:53.650835 1061 heartbeater.cc:500] Master 127.0.96.126:43817 was elected leader, sending a full tablet report...
I20261120 03:02:53.679777 1026 tablet_service.cc:1495] Processing CreateTablet for tablet 43123be7b9fc4ec2b0e23685d1605874 (DEFAULT_TABLE table=test-table [id=6e54636ea85345349db164550c1c7667]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:02:53.680243 1101 tablet_service.cc:1495] Processing CreateTablet for tablet 43123be7b9fc4ec2b0e23685d1605874 (DEFAULT_TABLE table=test-table [id=6e54636ea85345349db164550c1c7667]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:02:53.680871 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43123be7b9fc4ec2b0e23685d1605874. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:53.681277 1101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43123be7b9fc4ec2b0e23685d1605874. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:53.691589 1149 tablet_bootstrap.cc:492] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c: Bootstrap starting.
I20261120 03:02:53.692672 1150 tablet_bootstrap.cc:492] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b: Bootstrap starting.
I20261120 03:02:53.696022 1149 tablet_bootstrap.cc:654] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:53.696910 1150 tablet_bootstrap.cc:654] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:53.700207 1149 tablet_bootstrap.cc:492] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c: No bootstrap required, opened a new log
I20261120 03:02:53.700520 1150 tablet_bootstrap.cc:492] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b: No bootstrap required, opened a new log
I20261120 03:02:53.700572 1149 ts_tablet_manager.cc:1449] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20261120 03:02:53.700865 1150 ts_tablet_manager.cc:1449] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20261120 03:02:53.702297 1149 raft_consensus.cc:366] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.702448 1150 raft_consensus.cc:366] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.702750 1149 raft_consensus.cc:787] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e1c705ec064e7eb74a40c9a99fd34c, State: Initialized, Role: FOLLOWER
I20261120 03:02:53.702919 1150 raft_consensus.cc:787] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f8f09810044404b290ce30d0f28d4b, State: Initialized, Role: FOLLOWER
I20261120 03:02:53.703248 1149 consensus_queue.cc:261] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.703392 1150 consensus_queue.cc:261] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.705753 1149 ts_tablet_manager.cc:1480] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20261120 03:02:53.706166 1136 heartbeater.cc:500] Master 127.0.96.126:43817 was elected leader, sending a full tablet report...
I20261120 03:02:53.706687 1150 ts_tablet_manager.cc:1480] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20261120 03:02:53.734054 385 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20261120 03:02:53.734243 385 raft_consensus.cc:433] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 0 FOLLOWER]: Emulating election...
I20261120 03:02:53.734408 385 raft_consensus.cc:3296] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:53.735525 385 raft_consensus.cc:744] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 1 LEADER]: Becoming Leader. State: Replica: a4e1c705ec064e7eb74a40c9a99fd34c, State: Running, Role: LEADER
W20261120 03:02:53.736013 1061 heartbeater.cc:648] Failed to heartbeat to 127.0.96.126:43817 (0 consecutive failures): IO error: failing all heartbeats for tests
I20261120 03:02:53.736195 385 consensus_queue.cc:238] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.740449 385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:53.750219 1111 raft_consensus.cc:3296] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:53.757679 1156 consensus_queue.cc:1049] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:02:53.760089 385 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20261120 03:02:53.865373 385 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20261120 03:02:53.865636 385 raft_consensus.cc:433] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 1 LEADER]: Emulating election...
I20261120 03:02:53.865844 385 raft_consensus.cc:3291] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:02:53.866024 385 raft_consensus.cc:787] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4e1c705ec064e7eb74a40c9a99fd34c, State: Running, Role: LEADER
I20261120 03:02:53.866446 385 consensus_queue.cc:261] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.867033 385 raft_consensus.cc:3296] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:02:53.867514 385 raft_consensus.cc:744] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 2 LEADER]: Becoming Leader. State: Replica: a4e1c705ec064e7eb74a40c9a99fd34c, State: Running, Role: LEADER
I20261120 03:02:53.868039 385 consensus_queue.cc:238] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.872304 385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:53.873689 1111 raft_consensus.cc:3296] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:02:53.874851 1111 raft_consensus.cc:1338] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 2 FOLLOWER]: Refusing update from remote peer a4e1c705ec064e7eb74a40c9a99fd34c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261120 03:02:53.875900 385 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20261120 03:02:53.879434 1156 consensus_queue.cc:1049] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 03:02:53.980484 385 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20261120 03:02:53.980762 385 raft_consensus.cc:433] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 2 LEADER]: Emulating election...
I20261120 03:02:53.980983 385 raft_consensus.cc:3291] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 2 LEADER]: Stepping down as leader of term 2
I20261120 03:02:53.981207 385 raft_consensus.cc:787] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4e1c705ec064e7eb74a40c9a99fd34c, State: Running, Role: LEADER
I20261120 03:02:53.981604 385 consensus_queue.cc:261] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.982203 385 raft_consensus.cc:3296] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:02:53.982753 385 raft_consensus.cc:744] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 3 LEADER]: Becoming Leader. State: Replica: a4e1c705ec064e7eb74a40c9a99fd34c, State: Running, Role: LEADER
I20261120 03:02:53.983289 385 consensus_queue.cc:238] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.986923 385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:53.988339 1111 raft_consensus.cc:3296] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:02:53.989166 1111 raft_consensus.cc:1338] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 3 FOLLOWER]: Refusing update from remote peer a4e1c705ec064e7eb74a40c9a99fd34c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20261120 03:02:53.990523 385 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20261120 03:02:53.994103 1161 consensus_queue.cc:1049] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:02:54.097016 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:02:54.113597 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:02:54.114262 385 tablet_replica.cc:333] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c: stopping tablet replica
I20261120 03:02:54.114748 385 raft_consensus.cc:2440] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 3 LEADER]: Raft consensus shutting down.
I20261120 03:02:54.115278 385 raft_consensus.cc:2469] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 3 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:54.133369 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:02:54.139866 385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:02:54.152577 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:02:54.153245 385 tablet_replica.cc:333] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b: stopping tablet replica
I20261120 03:02:54.153752 385 raft_consensus.cc:2440] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 3 FOLLOWER]: Raft consensus shutting down.
I20261120 03:02:54.154105 385 raft_consensus.cc:2469] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 3 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:54.171200 385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:02:54.177789 385 master.cc:585] Master@127.0.96.126:43817 shutting down...
I20261120 03:02:54.190485 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:02:54.190889 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:54.191217 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db: stopping tablet replica
I20261120 03:02:54.207733 385 master.cc:607] Master@127.0.96.126:43817 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (981 ms)
[----------] 3 tests from LeadershipChangeReportingTest (5346 ms total)
[----------] 4 tests from TsTabletManagerITest
[ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20261120 03:02:54.229281 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:44799
I20261120 03:02:54.230238 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:54.234601 1166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.235781 1167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.237306 1169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:54.237586 385 server_base.cc:1061] running on GCE node
I20261120 03:02:54.238300 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:54.238488 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:54.238648 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143774238633 us; error 0 us; skew 500 ppm
I20261120 03:02:54.239130 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:54.241142 385 webserver.cc:577] Webserver started at http://127.0.96.126:35679/ using document root <none> and password file <none>
I20261120 03:02:54.241587 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:54.241773 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:54.242013 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:54.242929 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "13b572e8c8814f5e946a5e885cb2d475"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.246845 385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261120 03:02:54.249711 1174 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:54.250526 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 03:02:54.250730 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "13b572e8c8814f5e946a5e885cb2d475"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.250938 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:54.267541 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:54.268543 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:54.300067 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:44799
I20261120 03:02:54.300164 1225 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:44799 every 8 connection(s)
I20261120 03:02:54.303516 1226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:54.312043 1226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475: Bootstrap starting.
I20261120 03:02:54.315793 1226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:54.319000 1226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475: No bootstrap required, opened a new log
I20261120 03:02:54.320672 1226 raft_consensus.cc:366] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b572e8c8814f5e946a5e885cb2d475" member_type: VOTER }
I20261120 03:02:54.321025 1226 raft_consensus.cc:392] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:54.321254 1226 raft_consensus.cc:787] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b572e8c8814f5e946a5e885cb2d475, State: Initialized, Role: FOLLOWER
I20261120 03:02:54.321837 1226 consensus_queue.cc:261] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b572e8c8814f5e946a5e885cb2d475" member_type: VOTER }
I20261120 03:02:54.322263 1226 raft_consensus.cc:407] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:02:54.322497 1226 raft_consensus.cc:534] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:02:54.322732 1226 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:54.323709 1226 raft_consensus.cc:556] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b572e8c8814f5e946a5e885cb2d475" member_type: VOTER }
I20261120 03:02:54.327623 1226 leader_election.cc:304] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13b572e8c8814f5e946a5e885cb2d475; no voters:
I20261120 03:02:54.328213 1226 leader_election.cc:290] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:02:54.328286 1229 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:02:54.329769 1229 raft_consensus.cc:744] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 1 LEADER]: Becoming Leader. State: Replica: 13b572e8c8814f5e946a5e885cb2d475, State: Running, Role: LEADER
I20261120 03:02:54.330370 1229 consensus_queue.cc:238] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b572e8c8814f5e946a5e885cb2d475" member_type: VOTER }
I20261120 03:02:54.330857 1226 sys_catalog.cc:565] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:02:54.332979 1230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13b572e8c8814f5e946a5e885cb2d475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b572e8c8814f5e946a5e885cb2d475" member_type: VOTER } }
I20261120 03:02:54.333019 1231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13b572e8c8814f5e946a5e885cb2d475. Latest consensus state: current_term: 1 leader_uuid: "13b572e8c8814f5e946a5e885cb2d475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b572e8c8814f5e946a5e885cb2d475" member_type: VOTER } }
I20261120 03:02:54.333738 1231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:54.333770 1230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:54.343549 1235 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:02:54.347476 1235 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:02:54.348378 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:02:54.354766 1235 catalog_manager.cc:1398] Generated new cluster ID: 4b9ccdd3131b4e28abee56f1dc1a8219
I20261120 03:02:54.354969 1235 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:02:54.376744 1235 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:02:54.377746 1235 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:02:54.389850 1235 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475: Generated new TSK 0
I20261120 03:02:54.390295 1235 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:02:54.413851 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:54.419052 1247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.419752 1248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:54.421631 385 server_base.cc:1061] running on GCE node
W20261120 03:02:54.421535 1250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:54.422421 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:54.422636 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:54.422806 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143774422790 us; error 0 us; skew 500 ppm
I20261120 03:02:54.423281 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:54.425338 385 webserver.cc:577] Webserver started at http://127.0.96.65:36049/ using document root <none> and password file <none>
I20261120 03:02:54.425792 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:54.425993 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:54.426239 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:54.427279 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "fb70d1328b754ce9afa3cc8507fe336c"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.431437 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20261120 03:02:54.434484 1255 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:54.435268 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 03:02:54.435482 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "fb70d1328b754ce9afa3cc8507fe336c"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.435712 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:54.457656 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:54.458464 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:54.459717 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:54.461700 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:54.461869 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:54.462049 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:54.462173 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:54.495426 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:36933
I20261120 03:02:54.495503 1317 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:36933 every 8 connection(s)
I20261120 03:02:54.499506 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:54.505822 1322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.506392 1323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.508775 1326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:54.508876 385 server_base.cc:1061] running on GCE node
I20261120 03:02:54.509826 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:54.510123 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:54.510337 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143774510323 us; error 0 us; skew 500 ppm
I20261120 03:02:54.510998 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:54.511476 1318 heartbeater.cc:345] Connected to a master server at 127.0.96.126:44799
I20261120 03:02:54.511842 1318 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:54.512600 1318 heartbeater.cc:508] Master 127.0.96.126:44799 requested a full tablet report, sending...
I20261120 03:02:54.513527 385 webserver.cc:577] Webserver started at http://127.0.96.66:38411/ using document root <none> and password file <none>
I20261120 03:02:54.514070 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:54.514312 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:54.514532 1191 ts_manager.cc:210] Registered new tserver with Master: fb70d1328b754ce9afa3cc8507fe336c (127.0.96.65:36933)
I20261120 03:02:54.514621 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:54.515997 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.516628 1191 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33774
I20261120 03:02:54.520097 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20261120 03:02:54.523138 1330 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:54.523777 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 03:02:54.524014 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.524307 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:54.557129 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:54.557993 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:54.559185 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:54.561102 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:54.561272 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:54.561435 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:54.561555 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:54.595197 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:33381
I20261120 03:02:54.595307 1392 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:33381 every 8 connection(s)
I20261120 03:02:54.599495 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:54.605748 1396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.606714 1397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.609495 1399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:54.609799 385 server_base.cc:1061] running on GCE node
I20261120 03:02:54.610610 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:54.610821 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:54.611020 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143774611002 us; error 0 us; skew 500 ppm
I20261120 03:02:54.611620 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:54.612002 1393 heartbeater.cc:345] Connected to a master server at 127.0.96.126:44799
I20261120 03:02:54.612401 1393 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:54.613058 1393 heartbeater.cc:508] Master 127.0.96.126:44799 requested a full tablet report, sending...
I20261120 03:02:54.614274 385 webserver.cc:577] Webserver started at http://127.0.96.67:44599/ using document root <none> and password file <none>
I20261120 03:02:54.614737 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:54.614929 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:54.614892 1191 ts_manager.cc:210] Registered new tserver with Master: 83191dd6cf0a4e6ba32ec4d6b0baf8e1 (127.0.96.66:33381)
I20261120 03:02:54.615325 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:54.616328 1191 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33788
I20261120 03:02:54.616350 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-2-root/instance:
uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.620388 385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20261120 03:02:54.623044 1404 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:54.623750 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20261120 03:02:54.624014 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-2-root
uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.624347 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:54.633543 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:02:54.634411 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:54.635551 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:54.637517 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:54.637713 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:54.637936 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:54.638094 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:02:54.694826 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.67:46697
I20261120 03:02:54.694900 1466 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.67:46697 every 8 connection(s)
I20261120 03:02:54.707515 1467 heartbeater.cc:345] Connected to a master server at 127.0.96.126:44799
I20261120 03:02:54.707840 1467 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:54.708504 1467 heartbeater.cc:508] Master 127.0.96.126:44799 requested a full tablet report, sending...
I20261120 03:02:54.710311 1191 ts_manager.cc:210] Registered new tserver with Master: fd22d6a08b19497f8e0ba6f7730b2a1c (127.0.96.67:46697)
I20261120 03:02:54.710769 385 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011903127s
I20261120 03:02:54.711512 1191 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33794
I20261120 03:02:54.732864 1191 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20261120 03:02:54.734082 1191 catalog_manager.cc:7244] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261120 03:02:54.779484 1432 tablet_service.cc:1495] Processing CreateTablet for tablet b16a610a167240dca15b363f5ce2376f (DEFAULT_TABLE table=test-table [id=1f6fb7e133e443d480e0f8d0e95bf9f0]), partition=
I20261120 03:02:54.781028 1432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16a610a167240dca15b363f5ce2376f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:54.780997 1283 tablet_service.cc:1495] Processing CreateTablet for tablet b16a610a167240dca15b363f5ce2376f (DEFAULT_TABLE table=test-table [id=1f6fb7e133e443d480e0f8d0e95bf9f0]), partition=
I20261120 03:02:54.782060 1283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16a610a167240dca15b363f5ce2376f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:54.783007 1358 tablet_service.cc:1495] Processing CreateTablet for tablet b16a610a167240dca15b363f5ce2376f (DEFAULT_TABLE table=test-table [id=1f6fb7e133e443d480e0f8d0e95bf9f0]), partition=
I20261120 03:02:54.784435 1358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16a610a167240dca15b363f5ce2376f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:54.797142 1482 tablet_bootstrap.cc:492] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c: Bootstrap starting.
I20261120 03:02:54.799448 1483 tablet_bootstrap.cc:492] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c: Bootstrap starting.
I20261120 03:02:54.803227 1484 tablet_bootstrap.cc:492] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1: Bootstrap starting.
I20261120 03:02:54.806152 1482 tablet_bootstrap.cc:654] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:54.807794 1483 tablet_bootstrap.cc:654] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:54.808522 1484 tablet_bootstrap.cc:654] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:54.813802 1482 tablet_bootstrap.cc:492] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c: No bootstrap required, opened a new log
I20261120 03:02:54.814944 1482 ts_tablet_manager.cc:1449] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20261120 03:02:54.815907 1483 tablet_bootstrap.cc:492] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c: No bootstrap required, opened a new log
I20261120 03:02:54.816356 1483 ts_tablet_manager.cc:1449] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20261120 03:02:54.818130 1484 tablet_bootstrap.cc:492] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1: No bootstrap required, opened a new log
I20261120 03:02:54.817884 1482 raft_consensus.cc:366] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:54.818554 1484 ts_tablet_manager.cc:1449] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20261120 03:02:54.818588 1482 raft_consensus.cc:392] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:54.818959 1482 raft_consensus.cc:787] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd22d6a08b19497f8e0ba6f7730b2a1c, State: Initialized, Role: FOLLOWER
I20261120 03:02:54.819274 1483 raft_consensus.cc:366] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:54.819559 1482 consensus_queue.cc:261] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:54.820026 1483 raft_consensus.cc:392] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:54.820377 1483 raft_consensus.cc:787] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb70d1328b754ce9afa3cc8507fe336c, State: Initialized, Role: FOLLOWER
I20261120 03:02:54.821105 1483 consensus_queue.cc:261] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:54.822284 1467 heartbeater.cc:500] Master 127.0.96.126:44799 was elected leader, sending a full tablet report...
I20261120 03:02:54.821482 1484 raft_consensus.cc:366] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:54.823068 1482 ts_tablet_manager.cc:1480] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20261120 03:02:54.823046 1484 raft_consensus.cc:392] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:54.823529 1484 raft_consensus.cc:787] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83191dd6cf0a4e6ba32ec4d6b0baf8e1, State: Initialized, Role: FOLLOWER
I20261120 03:02:54.825052 1318 heartbeater.cc:500] Master 127.0.96.126:44799 was elected leader, sending a full tablet report...
I20261120 03:02:54.825153 1484 consensus_queue.cc:261] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:54.827224 1483 ts_tablet_manager.cc:1480] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c: Time spent starting tablet: real 0.011s user 0.006s sys 0.002s
I20261120 03:02:54.829017 1393 heartbeater.cc:500] Master 127.0.96.126:44799 was elected leader, sending a full tablet report...
I20261120 03:02:54.829488 1484 ts_tablet_manager.cc:1480] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1: Time spent starting tablet: real 0.011s user 0.001s sys 0.008s
I20261120 03:02:55.010123 1488 raft_consensus.cc:534] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:02:55.010795 1488 raft_consensus.cc:556] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:55.012936 1488 leader_election.cc:290] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb70d1328b754ce9afa3cc8507fe336c (127.0.96.65:36933), 83191dd6cf0a4e6ba32ec4d6b0baf8e1 (127.0.96.66:33381)
I20261120 03:02:55.021757 1293 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "b16a610a167240dca15b363f5ce2376f" candidate_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb70d1328b754ce9afa3cc8507fe336c" is_pre_election: true
I20261120 03:02:55.022501 1293 raft_consensus.cc:2062] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate fd22d6a08b19497f8e0ba6f7730b2a1c in term 0
I20261120 03:02:55.023605 1368 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "b16a610a167240dca15b363f5ce2376f" candidate_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" is_pre_election: true
I20261120 03:02:55.023746 1405 leader_election.cc:304] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fb70d1328b754ce9afa3cc8507fe336c, fd22d6a08b19497f8e0ba6f7730b2a1c; no voters:
I20261120 03:02:55.024391 1368 raft_consensus.cc:2062] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate fd22d6a08b19497f8e0ba6f7730b2a1c in term 0
I20261120 03:02:55.024446 1488 raft_consensus.cc:3024] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:02:55.024843 1488 raft_consensus.cc:534] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:02:55.025144 1488 raft_consensus.cc:3296] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:55.026297 1488 raft_consensus.cc:556] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:55.031213 1488 leader_election.cc:290] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [CANDIDATE]: Term 1 election: Requested vote from peers fb70d1328b754ce9afa3cc8507fe336c (127.0.96.65:36933), 83191dd6cf0a4e6ba32ec4d6b0baf8e1 (127.0.96.66:33381)
I20261120 03:02:55.032352 1293 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "b16a610a167240dca15b363f5ce2376f" candidate_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb70d1328b754ce9afa3cc8507fe336c"
I20261120 03:02:55.032431 1368 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "b16a610a167240dca15b363f5ce2376f" candidate_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1"
I20261120 03:02:55.032927 1293 raft_consensus.cc:3296] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:55.033008 1368 raft_consensus.cc:3296] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:55.033754 1293 raft_consensus.cc:2062] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate fd22d6a08b19497f8e0ba6f7730b2a1c in term 1
I20261120 03:02:55.033829 1368 raft_consensus.cc:2062] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate fd22d6a08b19497f8e0ba6f7730b2a1c in term 1
I20261120 03:02:55.038827 1405 leader_election.cc:304] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fb70d1328b754ce9afa3cc8507fe336c, fd22d6a08b19497f8e0ba6f7730b2a1c; no voters:
I20261120 03:02:55.039443 1488 raft_consensus.cc:3024] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:02:55.039865 1488 raft_consensus.cc:744] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 1 LEADER]: Becoming Leader. State: Replica: fd22d6a08b19497f8e0ba6f7730b2a1c, State: Running, Role: LEADER
I20261120 03:02:55.040616 1488 consensus_queue.cc:238] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:55.047194 1488 consensus_queue.cc:1049] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:02:55.048202 1492 consensus_queue.cc:1049] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:02:55.048311 1191 catalog_manager.cc:5856] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c reported cstate change: term changed from 0 to 1, leader changed from <none> to fd22d6a08b19497f8e0ba6f7730b2a1c (127.0.96.67). New cstate: current_term: 1 leader_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:00.121285 385 tablet_replica.cc:333] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c: stopping tablet replica
I20261120 03:03:00.121924 385 raft_consensus.cc:2440] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 1 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:00.122460 385 raft_consensus.cc:2469] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:00.373221 1405 consensus_queue.cc:580] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [LEADER]: The tablet replica hosted on peer fb70d1328b754ce9afa3cc8507fe336c has failed
W20261120 03:03:00.373754 1405 consensus_peers.cc:597] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c -> Peer fb70d1328b754ce9afa3cc8507fe336c (127.0.96.65:36933): Couldn't send request to peer fb70d1328b754ce9afa3cc8507fe336c. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:00.390484 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:00.412593 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:00.429608 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:00.438791 385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:00.455166 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:00.455706 385 tablet_replica.cc:333] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1: stopping tablet replica
I20261120 03:03:00.456195 385 raft_consensus.cc:2440] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:00.456574 385 raft_consensus.cc:2469] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:00.473994 385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:00.482470 385 tablet_server.cc:179] TabletServer@127.0.96.67:0 shutting down...
I20261120 03:03:00.499900 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:00.500712 385 tablet_replica.cc:333] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c: stopping tablet replica
I20261120 03:03:00.501191 385 raft_consensus.cc:2440] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:00.501853 385 raft_consensus.cc:2469] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:00.518859 385 tablet_server.cc:196] TabletServer@127.0.96.67:0 shutdown complete.
I20261120 03:03:00.526975 385 master.cc:585] Master@127.0.96.126:44799 shutting down...
I20261120 03:03:00.544088 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:00.544662 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:00.544971 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475: stopping tablet replica
I20261120 03:03:00.561723 385 master.cc:607] Master@127.0.96.126:44799 shutdown complete.
[ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (6362 ms)
[ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20261120 03:03:00.591781 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:38541
I20261120 03:03:00.592882 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:00.597900 1509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:00.598313 1510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:00.599647 1512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:00.600333 385 server_base.cc:1061] running on GCE node
I20261120 03:03:00.601065 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:00.601217 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:00.601384 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143780601359 us; error 0 us; skew 500 ppm
I20261120 03:03:00.601859 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:00.603868 385 webserver.cc:577] Webserver started at http://127.0.96.126:37539/ using document root <none> and password file <none>
I20261120 03:03:00.604403 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:00.604562 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:00.604809 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:00.605844 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "c9caea48f0ac4663b52f5b498561e813"
format_stamp: "Formatted at 2026-11-20 03:03:00 on dist-test-slave-54pf"
I20261120 03:03:00.609854 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20261120 03:03:00.612886 1517 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:00.613592 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 03:03:00.613847 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "c9caea48f0ac4663b52f5b498561e813"
format_stamp: "Formatted at 2026-11-20 03:03:00 on dist-test-slave-54pf"
I20261120 03:03:00.614109 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:00.635443 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:00.636543 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:00.670980 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:38541
I20261120 03:03:00.671065 1568 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:38541 every 8 connection(s)
I20261120 03:03:00.674728 1569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:00.684383 1569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813: Bootstrap starting.
I20261120 03:03:00.688879 1569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:00.692813 1569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813: No bootstrap required, opened a new log
I20261120 03:03:00.694752 1569 raft_consensus.cc:366] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9caea48f0ac4663b52f5b498561e813" member_type: VOTER }
I20261120 03:03:00.695129 1569 raft_consensus.cc:392] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:00.695348 1569 raft_consensus.cc:787] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9caea48f0ac4663b52f5b498561e813, State: Initialized, Role: FOLLOWER
I20261120 03:03:00.695856 1569 consensus_queue.cc:261] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9caea48f0ac4663b52f5b498561e813" member_type: VOTER }
I20261120 03:03:00.696308 1569 raft_consensus.cc:407] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:00.696532 1569 raft_consensus.cc:534] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:00.696786 1569 raft_consensus.cc:3296] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:00.697757 1569 raft_consensus.cc:556] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9caea48f0ac4663b52f5b498561e813" member_type: VOTER }
I20261120 03:03:00.701982 1569 leader_election.cc:304] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9caea48f0ac4663b52f5b498561e813; no voters:
I20261120 03:03:00.702471 1569 leader_election.cc:290] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:00.702559 1572 raft_consensus.cc:3024] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:00.703918 1572 raft_consensus.cc:744] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 1 LEADER]: Becoming Leader. State: Replica: c9caea48f0ac4663b52f5b498561e813, State: Running, Role: LEADER
I20261120 03:03:00.704583 1572 consensus_queue.cc:238] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9caea48f0ac4663b52f5b498561e813" member_type: VOTER }
I20261120 03:03:00.705353 1569 sys_catalog.cc:565] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:00.706705 1573 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9caea48f0ac4663b52f5b498561e813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9caea48f0ac4663b52f5b498561e813" member_type: VOTER } }
I20261120 03:03:00.707118 1573 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:00.707831 1574 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9caea48f0ac4663b52f5b498561e813. Latest consensus state: current_term: 1 leader_uuid: "c9caea48f0ac4663b52f5b498561e813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9caea48f0ac4663b52f5b498561e813" member_type: VOTER } }
I20261120 03:03:00.708557 1574 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:00.710824 1576 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:00.715314 1576 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:00.720172 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:00.723467 1576 catalog_manager.cc:1398] Generated new cluster ID: 8c829e260350433bb29d36fa6ca10d3d
I20261120 03:03:00.723735 1576 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:00.737931 1576 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:00.739041 1576 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:00.752295 1576 catalog_manager.cc:6237] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813: Generated new TSK 0
I20261120 03:03:00.752805 1576 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:00.786230 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:00.791600 1590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:00.792630 1591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:00.794157 1593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:00.794343 385 server_base.cc:1061] running on GCE node
I20261120 03:03:00.795176 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:00.795384 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:00.795560 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143780795541 us; error 0 us; skew 500 ppm
I20261120 03:03:00.796051 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:00.797953 385 webserver.cc:577] Webserver started at http://127.0.96.65:34477/ using document root <none> and password file <none>
I20261120 03:03:00.798414 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:00.798621 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:00.798851 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:00.799840 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "510ca5f2907b4b9e8d66c9e553be512c"
format_stamp: "Formatted at 2026-11-20 03:03:00 on dist-test-slave-54pf"
I20261120 03:03:00.803959 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20261120 03:03:00.806663 1598 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:00.807471 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 03:03:00.807756 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "510ca5f2907b4b9e8d66c9e553be512c"
format_stamp: "Formatted at 2026-11-20 03:03:00 on dist-test-slave-54pf"
I20261120 03:03:00.808044 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:00.823899 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:00.824952 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:00.826256 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:00.828415 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:00.828616 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:00.828858 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:00.829046 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:00.866124 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:37041
I20261120 03:03:00.866219 1660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:37041 every 8 connection(s)
I20261120 03:03:00.887826 1661 heartbeater.cc:345] Connected to a master server at 127.0.96.126:38541
I20261120 03:03:00.888103 1661 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:00.888728 1661 heartbeater.cc:508] Master 127.0.96.126:38541 requested a full tablet report, sending...
I20261120 03:03:00.890677 1534 ts_manager.cc:210] Registered new tserver with Master: 510ca5f2907b4b9e8d66c9e553be512c (127.0.96.65:37041)
I20261120 03:03:00.890942 385 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.021797183s
I20261120 03:03:00.892396 1534 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35694
I20261120 03:03:00.908631 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:00.921871 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:00.936878 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:00.943432 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261120 03:03:00.945909 385 tablet_server.cc:94] deduplicated master addresses: 127.0.96.126:38541
W20261120 03:03:00.948938 1671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:00.949092 1672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:00.950181 1674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:00.951167 385 server_base.cc:1061] running on GCE node
I20261120 03:03:00.951890 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:00.952099 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:00.952296 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143780952284 us; error 0 us; skew 500 ppm
I20261120 03:03:00.952708 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:00.954696 385 webserver.cc:577] Webserver started at http://127.0.96.65:34477/ using document root <none> and password file <none>
I20261120 03:03:00.955101 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:00.955242 385 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261120 03:03:00.958850 385 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20261120 03:03:00.961122 1679 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:00.961823 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20261120 03:03:00.962112 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "510ca5f2907b4b9e8d66c9e553be512c"
format_stamp: "Formatted at 2026-11-20 03:03:00 on dist-test-slave-54pf"
I20261120 03:03:00.962409 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:00.976797 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:00.977682 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:00.978928 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:00.980959 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:00.981195 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:00.981401 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:00.981571 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:01.015816 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:37041
I20261120 03:03:01.015898 1741 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:37041 every 8 connection(s)
I20261120 03:03:01.023146 385 tablet_server.cc:179] TabletServer@127.0.96.65:37041 shutting down...
I20261120 03:03:01.032740 1742 heartbeater.cc:345] Connected to a master server at 127.0.96.126:38541
I20261120 03:03:01.033221 1742 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.034045 1742 heartbeater.cc:508] Master 127.0.96.126:38541 requested a full tablet report, sending...
I20261120 03:03:01.036316 1534 ts_manager.cc:210] Re-registered known tserver with Master: 510ca5f2907b4b9e8d66c9e553be512c (127.0.96.65:37041)
I20261120 03:03:01.040264 1534 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35712
I20261120 03:03:01.042716 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:01.057706 385 tablet_server.cc:196] TabletServer@127.0.96.65:37041 shutdown complete.
I20261120 03:03:01.063872 385 master.cc:585] Master@127.0.96.126:38541 shutting down...
I20261120 03:03:01.075528 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:01.075894 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:01.076334 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813: stopping tablet replica
I20261120 03:03:01.093824 385 master.cc:607] Master@127.0.96.126:38541 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (521 ms)
[ RUN ] TsTabletManagerITest.TestTableStats
I20261120 03:03:01.113286 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:43725,127.0.96.125:41159,127.0.96.124:46415
I20261120 03:03:01.114382 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:01.118534 1749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:01.119565 1750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.120769 385 server_base.cc:1061] running on GCE node
W20261120 03:03:01.121533 1752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.122123 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:01.122313 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:01.122448 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143781122433 us; error 0 us; skew 500 ppm
I20261120 03:03:01.122884 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:01.124884 385 webserver.cc:577] Webserver started at http://127.0.96.126:38517/ using document root <none> and password file <none>
I20261120 03:03:01.125314 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:01.125489 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:01.125732 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:01.126621 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "764428dd82184375a0c80cbf46397e89"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.130623 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20261120 03:03:01.133477 1757 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:01.134092 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 03:03:01.134356 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "764428dd82184375a0c80cbf46397e89"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.134619 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:01.172819 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:01.173703 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:01.204849 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:43725
I20261120 03:03:01.204936 1808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:43725 every 8 connection(s)
I20261120 03:03:01.208185 1809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:01.208460 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:01.213310 1811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:01.214320 1812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.216320 1809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
W20261120 03:03:01.216814 1814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.217439 385 server_base.cc:1061] running on GCE node
I20261120 03:03:01.218364 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:01.218664 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:01.218957 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143781218931 us; error 0 us; skew 500 ppm
I20261120 03:03:01.219802 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:01.222317 385 webserver.cc:577] Webserver started at http://127.0.96.125:43061/ using document root <none> and password file <none>
I20261120 03:03:01.222898 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:01.223140 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:01.223419 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:01.224553 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root/instance:
uuid: "e68c557e0a6047f5956737cfd33fffb5"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.228746 1809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.229902 385 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
W20261120 03:03:01.231052 1761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.96.125:41159: connect: Connection refused (error 111)
I20261120 03:03:01.233788 1822 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20261120 03:03:01.234326 1809 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.96.125:41159: Network error: Client connection negotiation failed: client connection to 127.0.96.125:41159: connect: Connection refused (error 111)
I20261120 03:03:01.234531 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 03:03:01.234881 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root
uuid: "e68c557e0a6047f5956737cfd33fffb5"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.235140 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:01.253479 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:01.254280 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:01.285888 1809 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } attempt: 1
I20261120 03:03:01.286357 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.125:41159
I20261120 03:03:01.286491 1873 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.125:41159 every 8 connection(s)
I20261120 03:03:01.290812 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261120 03:03:01.290877 1875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20261120 03:03:01.297047 1877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:01.298614 1878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.300428 1809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.301977 1875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.302690 385 server_base.cc:1061] running on GCE node
W20261120 03:03:01.303306 1880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.304421 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:01.304821 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:01.305078 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143781305058 us; error 0 us; skew 500 ppm
I20261120 03:03:01.305842 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20261120 03:03:01.307145 1809 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.96.124:46415: Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
I20261120 03:03:01.308540 385 webserver.cc:577] Webserver started at http://127.0.96.124:36403/ using document root <none> and password file <none>
I20261120 03:03:01.309111 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:01.309319 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:01.309609 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:01.310768 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root/instance:
uuid: "3a4598f24f6246008d66f45cff9355af"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.314247 1875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.316304 385 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20261120 03:03:01.320317 1887 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:01.321255 385 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20261120 03:03:01.321619 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root
uuid: "3a4598f24f6246008d66f45cff9355af"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.321916 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:01.324059 1875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } has no permanent_uuid. Determining permanent_uuid...
W20261120 03:03:01.329483 1875 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.96.124:46415: Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
I20261120 03:03:01.335871 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:01.336994 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:01.347918 1875 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } attempt: 1
I20261120 03:03:01.352564 1809 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } attempt: 1
W20261120 03:03:01.353295 1875 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.96.124:46415: Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
W20261120 03:03:01.357419 1809 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.96.124:46415: Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
I20261120 03:03:01.374946 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.124:46415
I20261120 03:03:01.375042 1939 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.124:46415 every 8 connection(s)
I20261120 03:03:01.377769 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:01.378423 1940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:01.384023 1940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.394284 1940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.397851 1809 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } attempt: 2
I20261120 03:03:01.403720 1940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.407793 1875 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } attempt: 2
I20261120 03:03:01.417574 1809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: Bootstrap starting.
I20261120 03:03:01.421790 1940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: Bootstrap starting.
I20261120 03:03:01.422850 1809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:01.426923 1940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:01.427493 1875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: Bootstrap starting.
I20261120 03:03:01.427958 1809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: No bootstrap required, opened a new log
I20261120 03:03:01.431100 1940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: No bootstrap required, opened a new log
I20261120 03:03:01.430864 1809 raft_consensus.cc:366] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.431659 1809 raft_consensus.cc:392] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:01.431965 1809 raft_consensus.cc:787] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764428dd82184375a0c80cbf46397e89, State: Initialized, Role: FOLLOWER
I20261120 03:03:01.432710 1809 consensus_queue.cc:261] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.433588 1940 raft_consensus.cc:366] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.434165 1940 raft_consensus.cc:392] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:01.434300 1875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:01.434554 1940 raft_consensus.cc:787] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a4598f24f6246008d66f45cff9355af, State: Initialized, Role: FOLLOWER
I20261120 03:03:01.434718 1947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.435432 1947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.435432 1940 consensus_queue.cc:261] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.436347 1809 sys_catalog.cc:565] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:01.437691 1948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.438521 1948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.439214 1940 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:01.446386 1875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: No bootstrap required, opened a new log
I20261120 03:03:01.449472 1875 raft_consensus.cc:366] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.450191 1875 raft_consensus.cc:392] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:01.450466 1875 raft_consensus.cc:787] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68c557e0a6047f5956737cfd33fffb5, State: Initialized, Role: FOLLOWER
I20261120 03:03:01.451133 1875 consensus_queue.cc:261] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.457391 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20261120 03:03:01.459287 1965 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.459831 1965 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.462009 1875 sys_catalog.cc:565] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: configured and running, proceeding with master startup.
W20261120 03:03:01.462469 1969 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20261120 03:03:01.462754 1969 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20261120 03:03:01.466526 1972 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20261120 03:03:01.466873 1972 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20261120 03:03:01.472626 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20261120 03:03:01.473889 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:01.475297 1983 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
E20261120 03:03:01.475345 385 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261120 03:03:01.475687 1983 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20261120 03:03:01.480404 1984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:01.480737 1985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:01.482128 1987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.482893 385 server_base.cc:1061] running on GCE node
I20261120 03:03:01.483560 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:01.483754 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:01.483917 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143781483902 us; error 0 us; skew 500 ppm
I20261120 03:03:01.484443 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:01.486449 385 webserver.cc:577] Webserver started at http://127.0.96.65:44329/ using document root <none> and password file <none>
I20261120 03:03:01.486902 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:01.487088 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:01.487344 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:01.488432 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.492588 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20261120 03:03:01.495311 1992 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:01.495929 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 03:03:01.496245 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.496510 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:01.515136 1947 raft_consensus.cc:534] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:01.515570 1947 raft_consensus.cc:556] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.517510 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:01.517567 1947 leader_election.cc:290] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:01.518203 1849 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5" is_pre_election: true
I20261120 03:03:01.518958 1849 raft_consensus.cc:2062] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 764428dd82184375a0c80cbf46397e89 in term 0
I20261120 03:03:01.518705 1915 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:01.519145 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:01.519536 1915 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 764428dd82184375a0c80cbf46397e89 in term 0
I20261120 03:03:01.519924 1761 leader_election.cc:304] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 764428dd82184375a0c80cbf46397e89, e68c557e0a6047f5956737cfd33fffb5; no voters:
I20261120 03:03:01.520692 1947 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:01.521049 1947 raft_consensus.cc:534] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:01.521186 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:01.521420 1947 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:01.522930 1947 raft_consensus.cc:556] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.523677 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:01.523969 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:01.524252 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:01.524459 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:01.528769 1947 leader_election.cc:290] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 1 election: Requested vote from peers e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:01.529448 1849 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5"
I20261120 03:03:01.529816 1915 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af"
I20261120 03:03:01.529968 1849 raft_consensus.cc:3296] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:01.530480 1915 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:01.531128 1849 raft_consensus.cc:2062] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 764428dd82184375a0c80cbf46397e89 in term 1
I20261120 03:03:01.531443 1915 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 764428dd82184375a0c80cbf46397e89 in term 1
I20261120 03:03:01.538159 1760 leader_election.cc:304] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89; no voters:
I20261120 03:03:01.539101 1947 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:01.539546 1947 raft_consensus.cc:744] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 1 LEADER]: Becoming Leader. State: Replica: 764428dd82184375a0c80cbf46397e89, State: Running, Role: LEADER
I20261120 03:03:01.540349 1947 consensus_queue.cc:238] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.545004 1998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764428dd82184375a0c80cbf46397e89. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.545918 1998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:01.549432 2015 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:01.551254 2014 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261120 03:03:01.552515 1998 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:01.557214 1948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764428dd82184375a0c80cbf46397e89. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.557950 1948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.560458 1965 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764428dd82184375a0c80cbf46397e89. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.561151 1965 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.561973 2015 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:01.568854 1948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.569548 1948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.573836 1947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.575049 1947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:01.576162 2014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.577209 2014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:01.579603 1965 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.580376 1965 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.609448 2015 catalog_manager.cc:1398] Generated new cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:01.609782 2015 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:01.640226 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:40485
I20261120 03:03:01.640429 2067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:40485 every 8 connection(s)
I20261120 03:03:01.661283 2015 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:01.663110 2015 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:01.663178 2068 heartbeater.cc:345] Connected to a master server at 127.0.96.124:46415
I20261120 03:03:01.667663 2068 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.668567 2068 heartbeater.cc:508] Master 127.0.96.124:46415 requested a full tablet report, sending...
I20261120 03:03:01.671627 1905 ts_manager.cc:210] Registered new tserver with Master: 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:01.674073 2069 heartbeater.cc:345] Connected to a master server at 127.0.96.125:41159
I20261120 03:03:01.674381 2069 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.675002 2069 heartbeater.cc:508] Master 127.0.96.125:41159 requested a full tablet report, sending...
I20261120 03:03:01.677265 1839 ts_manager.cc:210] Registered new tserver with Master: 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:01.681150 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261120 03:03:01.683161 385 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20261120 03:03:01.685400 2070 heartbeater.cc:345] Connected to a master server at 127.0.96.126:43725
I20261120 03:03:01.685720 2070 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.686367 2070 heartbeater.cc:508] Master 127.0.96.126:43725 requested a full tablet report, sending...
I20261120 03:03:01.689021 1774 ts_manager.cc:210] Registered new tserver with Master: 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
W20261120 03:03:01.690783 2075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:01.697875 2076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.698508 385 server_base.cc:1061] running on GCE node
W20261120 03:03:01.699224 2078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.700006 2015 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: Generated new TSK 0
I20261120 03:03:01.700690 2015 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:01.700641 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:01.701041 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:01.701247 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143781701231 us; error 0 us; skew 500 ppm
I20261120 03:03:01.701924 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:01.709890 385 webserver.cc:577] Webserver started at http://127.0.96.66:35965/ using document root <none> and password file <none>
I20261120 03:03:01.710428 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:01.710634 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:01.710883 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:01.711920 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "a157015bf640485fb805159465a9ab1b"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.716527 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20261120 03:03:01.719484 2083 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:01.720330 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261120 03:03:01.720567 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "a157015bf640485fb805159465a9ab1b"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.720791 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:01.738719 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:01.739830 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:01.741143 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:01.743608 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:01.743829 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:01.744057 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:01.744309 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20261120 03:03:01.800279 1774 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41076
I20261120 03:03:01.822762 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:36367
I20261120 03:03:01.822841 2145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:36367 every 8 connection(s)
I20261120 03:03:01.847575 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261120 03:03:01.850085 385 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261120 03:03:01.859768 2153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.860879 2146 heartbeater.cc:345] Connected to a master server at 127.0.96.124:46415
I20261120 03:03:01.861263 2146 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.862094 2146 heartbeater.cc:508] Master 127.0.96.124:46415 requested a full tablet report, sending...
W20261120 03:03:01.864267 2154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.864868 1905 ts_manager.cc:210] Registered new tserver with Master: a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:01.869925 2147 heartbeater.cc:345] Connected to a master server at 127.0.96.125:41159
I20261120 03:03:01.870450 2147 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.872058 2147 heartbeater.cc:508] Master 127.0.96.125:41159 requested a full tablet report, sending...
I20261120 03:03:01.873178 2148 heartbeater.cc:345] Connected to a master server at 127.0.96.126:43725
I20261120 03:03:01.873575 2148 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.873746 385 server_base.cc:1061] running on GCE node
I20261120 03:03:01.874464 2148 heartbeater.cc:508] Master 127.0.96.126:43725 requested a full tablet report, sending...
I20261120 03:03:01.875304 1839 ts_manager.cc:210] Registered new tserver with Master: a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
W20261120 03:03:01.875456 2156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.876552 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20261120 03:03:01.876696 1774 ts_manager.cc:210] Registered new tserver with Master: a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
W20261120 03:03:01.876863 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:01.877101 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143781877083 us; error 0 us; skew 500 ppm
I20261120 03:03:01.877864 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:01.878427 1774 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41092
I20261120 03:03:01.881500 385 webserver.cc:577] Webserver started at http://127.0.96.67:42435/ using document root <none> and password file <none>
I20261120 03:03:01.881984 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:01.882174 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:01.882433 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:01.883646 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-2-root/instance:
uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.888072 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20261120 03:03:01.891036 2161 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:01.891723 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 03:03:01.892037 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-2-root
uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.892364 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:01.902702 2070 heartbeater.cc:500] Master 127.0.96.126:43725 was elected leader, sending a full tablet report...
I20261120 03:03:01.904440 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:01.905884 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:01.907900 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:01.909979 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:01.910190 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:01.910494 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:01.910652 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:01.981515 2148 heartbeater.cc:500] Master 127.0.96.126:43725 was elected leader, sending a full tablet report...
I20261120 03:03:01.991431 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.67:35169
I20261120 03:03:01.991528 2223 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.67:35169 every 8 connection(s)
I20261120 03:03:02.014479 2224 heartbeater.cc:345] Connected to a master server at 127.0.96.124:46415
I20261120 03:03:02.014946 2224 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:02.015379 2225 heartbeater.cc:345] Connected to a master server at 127.0.96.125:41159
I20261120 03:03:02.015771 2225 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:02.015767 2224 heartbeater.cc:508] Master 127.0.96.124:46415 requested a full tablet report, sending...
I20261120 03:03:02.016768 2225 heartbeater.cc:508] Master 127.0.96.125:41159 requested a full tablet report, sending...
I20261120 03:03:02.019119 1905 ts_manager.cc:210] Registered new tserver with Master: 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.019138 1839 ts_manager.cc:210] Registered new tserver with Master: 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.020800 2226 heartbeater.cc:345] Connected to a master server at 127.0.96.126:43725
I20261120 03:03:02.021080 2226 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:02.021682 2226 heartbeater.cc:508] Master 127.0.96.126:43725 requested a full tablet report, sending...
I20261120 03:03:02.023357 1774 ts_manager.cc:210] Registered new tserver with Master: 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.023622 385 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019925065s
I20261120 03:03:02.025103 1774 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41106
I20261120 03:03:02.057925 1774 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20261120 03:03:02.060530 1774 catalog_manager.cc:7244] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261120 03:03:02.127462 2226 heartbeater.cc:500] Master 127.0.96.126:43725 was elected leader, sending a full tablet report...
I20261120 03:03:02.151404 2030 tablet_service.cc:1495] Processing CreateTablet for tablet 791003cb42cf4e50a7666946f7ffad6b (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.152942 2030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 791003cb42cf4e50a7666946f7ffad6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.153169 2189 tablet_service.cc:1495] Processing CreateTablet for tablet 791003cb42cf4e50a7666946f7ffad6b (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.154213 2188 tablet_service.cc:1495] Processing CreateTablet for tablet ff883ba4b464463a9628eff440f1190a (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.154630 2189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 791003cb42cf4e50a7666946f7ffad6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.155472 2188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff883ba4b464463a9628eff440f1190a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.155555 2029 tablet_service.cc:1495] Processing CreateTablet for tablet ff883ba4b464463a9628eff440f1190a (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.157862 2029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff883ba4b464463a9628eff440f1190a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.151424 2028 tablet_service.cc:1495] Processing CreateTablet for tablet 178ee4361c534583b29d93a1e67d3c4f (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.158679 2187 tablet_service.cc:1495] Processing CreateTablet for tablet 178ee4361c534583b29d93a1e67d3c4f (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.162472 2187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178ee4361c534583b29d93a1e67d3c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.163736 2028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178ee4361c534583b29d93a1e67d3c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.163647 2111 tablet_service.cc:1495] Processing CreateTablet for tablet 791003cb42cf4e50a7666946f7ffad6b (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.165644 2110 tablet_service.cc:1495] Processing CreateTablet for tablet ff883ba4b464463a9628eff440f1190a (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.167827 2111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 791003cb42cf4e50a7666946f7ffad6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.167251 2109 tablet_service.cc:1495] Processing CreateTablet for tablet 178ee4361c534583b29d93a1e67d3c4f (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.170498 2110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff883ba4b464463a9628eff440f1190a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.171813 2109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178ee4361c534583b29d93a1e67d3c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.205672 2246 tablet_bootstrap.cc:492] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: Bootstrap starting.
I20261120 03:03:02.208734 2245 tablet_bootstrap.cc:492] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: Bootstrap starting.
I20261120 03:03:02.229111 2245 tablet_bootstrap.cc:654] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.231546 2247 tablet_bootstrap.cc:492] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Bootstrap starting.
I20261120 03:03:02.233209 2246 tablet_bootstrap.cc:654] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.238560 2245 tablet_bootstrap.cc:492] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: No bootstrap required, opened a new log
I20261120 03:03:02.239066 2245 ts_tablet_manager.cc:1449] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.013s
I20261120 03:03:02.242333 2245 raft_consensus.cc:366] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.244211 2245 raft_consensus.cc:392] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.244576 2245 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.245257 2245 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.246409 2247 tablet_bootstrap.cc:654] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.247907 2246 tablet_bootstrap.cc:492] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: No bootstrap required, opened a new log
I20261120 03:03:02.248394 2246 ts_tablet_manager.cc:1449] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.014s
I20261120 03:03:02.251597 2245 ts_tablet_manager.cc:1480] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent starting tablet: real 0.012s user 0.003s sys 0.007s
I20261120 03:03:02.251835 2246 raft_consensus.cc:366] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.252869 2246 raft_consensus.cc:392] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.253268 2246 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.253899 2246 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.257562 2246 ts_tablet_manager.cc:1480] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20261120 03:03:02.260514 2246 tablet_bootstrap.cc:492] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: Bootstrap starting.
I20261120 03:03:02.264756 2245 tablet_bootstrap.cc:492] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: Bootstrap starting.
I20261120 03:03:02.272576 2246 tablet_bootstrap.cc:654] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.274636 2245 tablet_bootstrap.cc:654] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.285485 2247 tablet_bootstrap.cc:492] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: No bootstrap required, opened a new log
I20261120 03:03:02.286171 2247 ts_tablet_manager.cc:1449] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.009s
I20261120 03:03:02.286463 2246 tablet_bootstrap.cc:492] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: No bootstrap required, opened a new log
I20261120 03:03:02.287300 2246 ts_tablet_manager.cc:1449] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.005s
I20261120 03:03:02.285481 2245 tablet_bootstrap.cc:492] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: No bootstrap required, opened a new log
I20261120 03:03:02.288419 2245 ts_tablet_manager.cc:1449] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.010s
I20261120 03:03:02.289464 2247 raft_consensus.cc:366] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.290561 2247 raft_consensus.cc:392] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.289939 2246 raft_consensus.cc:366] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.290889 2247 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.291074 2246 raft_consensus.cc:392] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.291276 2245 raft_consensus.cc:366] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.291546 2246 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.291893 2245 raft_consensus.cc:392] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.291652 2247 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.292248 2245 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.292478 2246 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.292948 2245 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.294322 2250 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.294824 2250 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.300997 2246 ts_tablet_manager.cc:1480] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: Time spent starting tablet: real 0.013s user 0.004s sys 0.003s
I20261120 03:03:02.301820 2252 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.302469 2252 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.318480 2250 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.319551 2250 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.320734 2250 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.321342 2252 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.323341 2252 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.324342 2252 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.323707 2250 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.341189 2246 tablet_bootstrap.cc:492] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: Bootstrap starting.
I20261120 03:03:02.357268 2247 ts_tablet_manager.cc:1480] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Time spent starting tablet: real 0.071s user 0.016s sys 0.006s
I20261120 03:03:02.362228 2247 tablet_bootstrap.cc:492] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Bootstrap starting.
I20261120 03:03:02.377918 2252 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.383865 2247 tablet_bootstrap.cc:654] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.385218 2246 tablet_bootstrap.cc:654] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.385598 2255 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.417294 2042 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:02.418432 2042 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 0
I20261120 03:03:02.415817 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:02.419883 2086 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters:
I20261120 03:03:02.420099 2043 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 0
I20261120 03:03:02.420948 2259 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:02.421762 2259 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:02.422125 2259 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.422364 2245 ts_tablet_manager.cc:1480] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent starting tablet: real 0.134s user 0.031s sys 0.049s
I20261120 03:03:02.422947 2259 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.423215 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.397625 2255 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.425644 2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.426499 2198 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 0
I20261120 03:03:02.429916 2086 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters:
W20261120 03:03:02.432482 2085 leader_election.cc:343] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169): Illegal state: must be running to vote when last-logged opid is not known
I20261120 03:03:02.423431 2245 tablet_bootstrap.cc:492] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: Bootstrap starting.
I20261120 03:03:02.508622 2259 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.508270 1983 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:02.458779 2252 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:02.439563 2255 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:02.546509 1972 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:02.608533 2245 tablet_bootstrap.cc:700] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent opening tablet: real 0.154s user 0.003s sys 0.000s
I20261120 03:03:02.619151 1983 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: loading cluster ID for follower catalog manager: success
I20261120 03:03:02.620496 2252 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:02.623153 1972 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: loading cluster ID for follower catalog manager: success
I20261120 03:03:02.624706 2245 tablet_bootstrap.cc:654] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.625242 2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.633216 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:02.634094 2199 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.636853 2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.637665 2197 raft_consensus.cc:2628] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in current term 1: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:02.631693 2252 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.645912 2246 tablet_bootstrap.cc:492] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: No bootstrap required, opened a new log
I20261120 03:03:02.635952 2199 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 1
I20261120 03:03:02.656826 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:02.657378 2043 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.657281 2085 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters:
I20261120 03:03:02.657758 2247 tablet_bootstrap.cc:492] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: No bootstrap required, opened a new log
I20261120 03:03:02.659268 2247 ts_tablet_manager.cc:1449] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Time spent bootstrapping tablet: real 0.297s user 0.019s sys 0.038s
I20261120 03:03:02.659410 2262 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:02.659521 2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 1
I20261120 03:03:02.660004 2262 raft_consensus.cc:744] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Becoming Leader. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:02.661204 2262 consensus_queue.cc:238] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.654681 2246 ts_tablet_manager.cc:1449] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: Time spent bootstrapping tablet: real 0.314s user 0.011s sys 0.023s
I20261120 03:03:02.667680 2266 raft_consensus.cc:534] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:02.668306 2266 raft_consensus.cc:556] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:02.669775 2247 raft_consensus.cc:366] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.670156 2246 raft_consensus.cc:366] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.670531 2247 raft_consensus.cc:392] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.670904 2246 raft_consensus.cc:392] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.671357 2266 leader_election.cc:290] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159)
I20261120 03:03:02.671458 1849 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5" is_pre_election: true
I20261120 03:03:02.672328 1849 raft_consensus.cc:2062] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 3a4598f24f6246008d66f45cff9355af in term 1
I20261120 03:03:02.671016 2247 raft_consensus.cc:787] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.677009 1891 leader_election.cc:304] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af, e68c557e0a6047f5956737cfd33fffb5; no voters:
I20261120 03:03:02.680531 2245 tablet_bootstrap.cc:492] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: No bootstrap required, opened a new log
I20261120 03:03:02.671308 2246 raft_consensus.cc:787] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.682585 1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:02.683976 2042 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:02.684707 2042 raft_consensus.cc:2628] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 1: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:02.686012 2246 consensus_queue.cc:261] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.688052 2246 ts_tablet_manager.cc:1480] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20261120 03:03:02.683529 2266 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:02.689410 2266 raft_consensus.cc:534] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 1 FOLLOWER]: Starting leader election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:02.689857 2266 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:02.690425 2245 ts_tablet_manager.cc:1449] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent bootstrapping tablet: real 0.267s user 0.025s sys 0.000s
I20261120 03:03:02.697743 2252 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.703401 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:02.708098 2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.709172 2121 raft_consensus.cc:2628] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in current term 1: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:02.709988 2245 raft_consensus.cc:366] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.703928 2043 raft_consensus.cc:2612] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader election vote request: already granted 'yes' vote to a157015bf640485fb805159465a9ab1b in term 1; re-sending same reply
I20261120 03:03:02.707741 2247 consensus_queue.cc:261] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.713672 2245 raft_consensus.cc:392] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.714031 2245 raft_consensus.cc:787] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.714797 2245 consensus_queue.cc:261] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.717041 2270 raft_consensus.cc:534] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:02.723614 2270 raft_consensus.cc:556] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:02.728578 2256 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.728994 2256 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.729643 2247 ts_tablet_manager.cc:1480] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Time spent starting tablet: real 0.070s user 0.009s sys 0.001s
I20261120 03:03:02.730747 2247 tablet_bootstrap.cc:492] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Bootstrap starting.
I20261120 03:03:02.724376 1972 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: acquiring CA information for follower catalog manager: success
I20261120 03:03:02.732581 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.734360 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:02.734506 2259 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.731182 2252 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.734980 2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.735447 2271 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20261120 03:03:02.747435 2245 ts_tablet_manager.cc:1480] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent starting tablet: real 0.050s user 0.008s sys 0.000s
I20261120 03:03:02.751380 2197 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 0
I20261120 03:03:02.752305 2255 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.745491 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.747028 2043 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.747208 2259 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.754987 2255 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.759454 1994 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters:
I20261120 03:03:02.756866 2043 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 1
I20261120 03:03:02.759476 2256 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:02.761622 1915 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:02.761929 1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:02.763000 2270 leader_election.cc:290] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:02.763828 2250 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:02.766490 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
W20261120 03:03:02.765003 1994 leader_election.cc:343] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169): Illegal state: must be running to vote when last-logged opid is not known
I20261120 03:03:02.765813 1983 catalog_manager.cc:1625] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: acquiring CA information for follower catalog manager: success
I20261120 03:03:02.765167 2250 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:02.763335 2266 raft_consensus.cc:556] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:02.771816 2259 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:02.773003 1772 catalog_manager.cc:5856] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b reported cstate change: term changed from 0 to 1, leader changed from <none> to a157015bf640485fb805159465a9ab1b (127.0.96.66). New cstate: current_term: 1 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:02.771870 2250 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.776104 1914 raft_consensus.cc:1303] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Rejecting Update request from peer 764428dd82184375a0c80cbf46397e89 for earlier term 1. Current term is 2. Ops: []
I20261120 03:03:02.771804 1825 leader_election.cc:304] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68c557e0a6047f5956737cfd33fffb5; no voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89
I20261120 03:03:02.792665 2250 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.795655 2270 raft_consensus.cc:2966] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261120 03:03:02.798923 1996 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b
I20261120 03:03:02.800289 2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.806499 2255 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:02.806785 1983 catalog_manager.cc:1653] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261120 03:03:02.808326 2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.809010 2197 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 0
I20261120 03:03:02.810130 2247 tablet_bootstrap.cc:654] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.813608 2269 consensus_queue.cc:1060] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 }, Status: INVALID_TERM, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20261120 03:03:02.814860 2269 raft_consensus.cc:3291] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:03:02.815279 2269 raft_consensus.cc:787] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 764428dd82184375a0c80cbf46397e89, State: Running, Role: LEADER
I20261120 03:03:02.816671 2269 consensus_queue.cc:261] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:02.817236 1996 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b
I20261120 03:03:02.817811 2269 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:02.818418 2277 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261120 03:03:02.819567 2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.821072 2274 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261120 03:03:02.822641 2247 tablet_bootstrap.cc:492] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: No bootstrap required, opened a new log
I20261120 03:03:02.823276 2247 ts_tablet_manager.cc:1449] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Time spent bootstrapping tablet: real 0.093s user 0.006s sys 0.010s
I20261120 03:03:02.823808 2086 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters:
I20261120 03:03:02.824765 2274 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:02.825171 2274 raft_consensus.cc:744] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Becoming Leader. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:02.825906 2274 consensus_queue.cc:238] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.827873 2258 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20261120 03:03:02.827920 2085 leader_election.cc:343] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169): Illegal state: must be running to vote when last-logged opid is not known
I20261120 03:03:02.829823 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:02.830956 2043 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 1: Already voted for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in this term.
I20261120 03:03:02.832476 1972 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261120 03:03:02.836035 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:02.836836 2043 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a157015bf640485fb805159465a9ab1b in current term 1: Already voted for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in this term.
I20261120 03:03:02.838817 2250 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 election: Requested vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:02.840457 2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:02.841037 2197 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.841897 2197 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 1
I20261120 03:03:02.843253 2121 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 0
I20261120 03:03:02.844588 2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:02.846408 2121 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.828962 2247 raft_consensus.cc:366] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.851075 2121 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 1
I20261120 03:03:02.850909 1994 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters:
I20261120 03:03:02.852660 2256 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:02.853150 2256 raft_consensus.cc:744] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 LEADER]: Becoming Leader. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:02.853250 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.853935 2256 consensus_queue.cc:238] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.852188 2247 raft_consensus.cc:392] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.858752 2247 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.862499 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.863821 2120 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 1: Already voted for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in this term.
I20261120 03:03:02.864691 2085 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:02.862715 2247 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.867949 2262 raft_consensus.cc:2984] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20261120 03:03:02.868577 2262 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:02.868947 2247 ts_tablet_manager.cc:1480] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Time spent starting tablet: real 0.045s user 0.007s sys 0.001s
I20261120 03:03:02.875973 2256 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
E20261120 03:03:02.886801 1772 catalog_manager.cc:6019] Error updating tablets from a157015bf640485fb805159465a9ab1b: Illegal state: Replica 764428dd82184375a0c80cbf46397e89 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }. Tablet report was: is_incremental: true updated_tablets { tablet_id: "ff883ba4b464463a9628eff440f1190a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } sequence_number: 9
I20261120 03:03:02.891506 2256 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261120 03:03:02.898138 2165 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:02.901292 2255 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261120 03:03:02.901666 2165 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:02.902486 2255 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261120 03:03:02.904680 2260 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.905138 2260 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.915202 2260 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:02.917703 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:02.937105 2266 leader_election.cc:290] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 2 election: Requested vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159)
I20261120 03:03:02.937829 2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.941951 1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89"
I20261120 03:03:02.942981 1784 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate 3a4598f24f6246008d66f45cff9355af in term 2
I20261120 03:03:02.944809 1849 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5"
I20261120 03:03:02.948226 2197 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.953605 2165 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:02.954973 2255 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20261120 03:03:02.964715 2232 master_proxy_rpc.cc:203] Re-attempting IsCreateTableDone request to leader Master (127.0.96.126:43725)
I20261120 03:03:02.971812 2274 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:02.977507 1888 leader_election.cc:304] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89; no voters: e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:02.978375 2266 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:02.978837 2266 raft_consensus.cc:744] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 LEADER]: Becoming Leader. State: Replica: 3a4598f24f6246008d66f45cff9355af, State: Running, Role: LEADER
I20261120 03:03:02.979403 2266 consensus_queue.cc:238] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:02.992273 1784 raft_consensus.cc:1338] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 2 FOLLOWER]: Refusing update from remote peer 3a4598f24f6246008d66f45cff9355af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20261120 03:03:02.994408 1849 raft_consensus.cc:3296] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:02.995379 1849 raft_consensus.cc:1338] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Refusing update from remote peer 3a4598f24f6246008d66f45cff9355af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20261120 03:03:02.994992 2292 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [LEADER]: Connected to new peer: Peer: permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20261120 03:03:03.000751 2269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a4598f24f6246008d66f45cff9355af. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.001415 2269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:03.007192 2291 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [LEADER]: Connected to new peer: Peer: permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20261120 03:03:03.008562 2275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a4598f24f6246008d66f45cff9355af. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.009217 2275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:03.012696 2266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.013383 2269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.013757 2266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:03.014021 2269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:03.025161 2295 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a4598f24f6246008d66f45cff9355af. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.025820 2295 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:03.026546 2270 raft_consensus.cc:534] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a4598f24f6246008d66f45cff9355af)
I20261120 03:03:03.027014 2294 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:03.026940 2270 raft_consensus.cc:556] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:03.030592 1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:03.031872 2297 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.032629 2297 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:03.033201 2270 leader_election.cc:290] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:03.034653 2294 catalog_manager.cc:694] Loaded metadata for table test-table [id=267fd8b91bac4f058896b2ba899401ba]
I20261120 03:03:03.042394 2294 tablet_loader.cc:96] loaded metadata for tablet 178ee4361c534583b29d93a1e67d3c4f (table test-table [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:03.043653 2294 tablet_loader.cc:96] loaded metadata for tablet 791003cb42cf4e50a7666946f7ffad6b (table test-table [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:03.044938 2294 tablet_loader.cc:96] loaded metadata for tablet ff883ba4b464463a9628eff440f1190a (table test-table [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:03.046310 2294 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:03.046729 1914 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:03.048518 1825 leader_election.cc:304] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68c557e0a6047f5956737cfd33fffb5; no voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89
I20261120 03:03:03.049153 2270 raft_consensus.cc:2966] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20261120 03:03:03.050232 2294 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:03.050529 2294 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:03.053799 2294 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:03.056787 2294 catalog_manager.cc:6248] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: Loaded TSK: 0
I20261120 03:03:03.057824 2294 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:03.080775 1905 catalog_manager.cc:5856] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b reported cstate change: term changed from 0 to 1, leader changed from <none> to a157015bf640485fb805159465a9ab1b (127.0.96.66). New cstate: current_term: 1 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:03.087255 2291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.087859 2291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:03.138780 1969 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261120 03:03:03.178218 2068 heartbeater.cc:500] Master 127.0.96.124:46415 was elected leader, sending a full tablet report...
I20261120 03:03:03.182754 1905 catalog_manager.cc:5856] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 0 to 1, leader changed from <none> to a157015bf640485fb805159465a9ab1b (127.0.96.66). New cstate: current_term: 1 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } }
I20261120 03:03:03.184288 1905 catalog_manager.cc:5856] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65). New cstate: current_term: 1 leader_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:03.199222 2146 heartbeater.cc:500] Master 127.0.96.124:46415 was elected leader, sending a full tablet report...
I20261120 03:03:03.203706 2224 heartbeater.cc:500] Master 127.0.96.124:46415 was elected leader, sending a full tablet report...
I20261120 03:03:03.426375 2042 tablet_service.cc:2056] Received LeaderStepDown RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b"
dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
from {username='slave'} at 127.0.0.1:49178
I20261120 03:03:03.426936 2042 raft_consensus.cc:635] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 LEADER]: Received request to step down
I20261120 03:03:03.427215 2042 raft_consensus.cc:3291] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:03:03.427433 2042 raft_consensus.cc:787] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:03.428148 2042 consensus_queue.cc:261] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.429173 2042 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:03.595358 2259 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:03.597456 2319 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:03.595988 2259 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.597941 2319 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.599843 2319 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.599972 2259 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:03.600638 2042 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:03.600934 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:03.601042 2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:03.601383 2042 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 2
I20261120 03:03:03.601042 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:03.602183 2121 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 1
I20261120 03:03:03.602244 2199 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 1
I20261120 03:03:03.601655 2043 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 2
I20261120 03:03:03.602716 2086 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters:
I20261120 03:03:03.603652 2259 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:03.604094 2259 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:03.603659 2165 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters:
I20261120 03:03:03.604499 2259 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:03.605387 2259 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.606396 2319 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:03.606786 2319 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:03.607127 2319 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:03.608353 2319 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.612389 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:03.612799 2259 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 2 election: Requested vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:03.613070 2199 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a157015bf640485fb805159465a9ab1b in current term 2: Already voted for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in this term.
I20261120 03:03:03.612936 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:03.614144 2043 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 2
I20261120 03:03:03.614933 2319 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.615767 2042 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:03.615844 2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:03.616712 2121 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 2: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:03.616672 2042 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 2: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:03.617955 2164 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:03.618654 2319 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261120 03:03:03.620864 2086 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c
I20261120 03:03:03.621533 2259 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:03.622031 2259 raft_consensus.cc:744] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 LEADER]: Becoming Leader. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:03.623245 2259 consensus_queue.cc:238] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.627712 2043 raft_consensus.cc:1338] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Refusing update from remote peer a157015bf640485fb805159465a9ab1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:03.627761 2199 raft_consensus.cc:1338] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Refusing update from remote peer a157015bf640485fb805159465a9ab1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:03.629319 2259 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:03.647727 1903 catalog_manager.cc:5856] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 1 to 2, leader changed from 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65) to a157015bf640485fb805159465a9ab1b (127.0.96.66). New cstate: current_term: 2 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } }
I20261120 03:03:03.648312 2274 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:03.663630 2121 tablet_service.cc:2056] Received LeaderStepDown RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b"
dest_uuid: "a157015bf640485fb805159465a9ab1b"
from {username='slave'} at 127.0.0.1:56228
I20261120 03:03:03.664348 2121 raft_consensus.cc:635] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 LEADER]: Received request to step down
I20261120 03:03:03.664721 2121 raft_consensus.cc:3291] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 LEADER]: Stepping down as leader of term 2
I20261120 03:03:03.665004 2121 raft_consensus.cc:787] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:03.665735 2121 consensus_queue.cc:261] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.666584 2121 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:03.868399 2320 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:03.868848 2320 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.871547 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:03.873900 2320 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.874303 2121 tablet_service.cc:2056] Received LeaderStepDown RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f"
dest_uuid: "a157015bf640485fb805159465a9ab1b"
from {username='slave'} at 127.0.0.1:56228
I20261120 03:03:03.874853 2121 raft_consensus.cc:635] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Received request to step down
I20261120 03:03:03.875174 2121 raft_consensus.cc:3291] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:03:03.875451 2121 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:03.876237 2121 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:03.877398 2256 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:03.877928 2256 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.879062 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:03.879755 2256 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.877440 2121 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:03.881052 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:03.879726 2120 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 3
I20261120 03:03:03.882232 2199 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 2
I20261120 03:03:03.883134 1994 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters:
I20261120 03:03:03.883859 2256 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261120 03:03:03.884244 2256 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:03.884622 2256 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:03.884730 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:03.885442 2120 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 3
I20261120 03:03:03.885470 2256 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.888784 2165 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:03.889612 2320 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261120 03:03:03.890007 2320 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Starting leader election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:03.890380 2320 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:03.891129 2320 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.892400 2256 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 election: Requested vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.893208 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:03.893445 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:03.893813 2199 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in current term 3: Already voted for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in this term.
I20261120 03:03:03.894569 2120 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 3
I20261120 03:03:03.897562 2320 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 3 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.898211 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:03.898895 2043 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 3: Already voted for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in this term.
I20261120 03:03:03.898900 2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:03.899704 2121 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 3: Already voted for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in this term.
I20261120 03:03:03.900411 1996 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c
I20261120 03:03:03.900616 2165 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:03.901129 2256 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Leader election won for term 3
I20261120 03:03:03.901352 2320 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20261120 03:03:03.901640 2256 raft_consensus.cc:744] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 LEADER]: Becoming Leader. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:03.902495 2256 consensus_queue.cc:238] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.907697 2121 raft_consensus.cc:1338] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261120 03:03:03.909394 2250 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:03.909962 2199 raft_consensus.cc:1338] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261120 03:03:03.911548 2250 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:03.911646 1905 catalog_manager.cc:5856] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 2 to 3, leader changed from a157015bf640485fb805159465a9ab1b (127.0.96.66) to 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65). New cstate: current_term: 3 leader_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:03.992897 2319 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:03.993376 2319 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:03.994890 2319 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.995886 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:03.996368 2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:03.996980 2121 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 2
I20261120 03:03:03.997798 2165 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:03.998508 2319 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:03.998811 2319 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:03.999080 2319 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:03.999629 2319 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.004513 2319 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:04.005592 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:04.005810 2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:04.006906 2121 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 2
I20261120 03:03:04.011565 2165 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:04.012323 2319 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:04.012821 2319 raft_consensus.cc:744] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Becoming Leader. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:04.014002 2319 consensus_queue.cc:238] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.018575 2121 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:04.020144 2043 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:04.021127 2043 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:04.023406 2320 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:04.023005 1903 catalog_manager.cc:5856] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c reported cstate change: term changed from 1 to 2, leader changed from a157015bf640485fb805159465a9ab1b (127.0.96.66) to 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67). New cstate: current_term: 2 leader_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:04.031750 2320 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:04.091207 2120 tablet_service.cc:2056] Received LeaderStepDown RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a"
dest_uuid: "a157015bf640485fb805159465a9ab1b"
from {username='slave'} at 127.0.0.1:56228
I20261120 03:03:04.091812 2120 raft_consensus.cc:635] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Received request to step down
I20261120 03:03:04.092204 2120 raft_consensus.cc:3291] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:03:04.092602 2120 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:04.094293 2120 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.095515 2120 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:04.234526 2319 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:04.236404 2319 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.239189 2319 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:04.239409 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:04.240289 2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 1
I20261120 03:03:04.240104 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:04.240926 2120 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 2
I20261120 03:03:04.241302 2164 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters:
I20261120 03:03:04.242024 2319 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:04.242267 2319 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:04.242478 2319 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:04.242990 2319 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.248483 2319 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:04.249133 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:04.249297 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:04.249583 2043 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:04.250294 2120 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 2
I20261120 03:03:04.250548 2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 2
I20261120 03:03:04.255501 2165 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters:
I20261120 03:03:04.256215 2319 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:04.256698 2319 raft_consensus.cc:744] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Becoming Leader. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:04.257380 2319 consensus_queue.cc:238] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.261626 2120 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:04.262068 2043 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:04.263878 2320 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:04.264539 2324 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:04.267948 1905 catalog_manager.cc:5856] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c reported cstate change: term changed from 1 to 2, leader changed from a157015bf640485fb805159465a9ab1b (127.0.96.66) to 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67). New cstate: current_term: 2 leader_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:04.322803 2199 tablet_service.cc:2056] Received LeaderStepDown RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f"
dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
from {username='slave'} at 127.0.0.1:52878
I20261120 03:03:04.323346 2199 raft_consensus.cc:635] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Received request to step down
I20261120 03:03:04.323707 2199 raft_consensus.cc:3291] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Stepping down as leader of term 2
I20261120 03:03:04.323993 2199 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:04.324757 2199 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.326014 2199 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:04.438484 2262 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:04.438977 2262 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.440690 2262 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:04.441308 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:04.441581 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:04.442328 2199 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 3
I20261120 03:03:04.443209 2085 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:04.443809 2262 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261120 03:03:04.444207 2262 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:04.444469 2262 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:04.445071 2262 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.451171 2262 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 3 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:04.452080 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:04.452495 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:04.453622 2199 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 3
I20261120 03:03:04.459064 2085 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:04.459654 2262 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader election won for term 3
I20261120 03:03:04.460071 2262 raft_consensus.cc:744] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 3 LEADER]: Becoming Leader. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:04.461123 2262 consensus_queue.cc:238] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.465384 2199 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Refusing update from remote peer a157015bf640485fb805159465a9ab1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261120 03:03:04.466370 2043 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:04.469219 2259 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:04.470098 2043 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Refusing update from remote peer a157015bf640485fb805159465a9ab1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261120 03:03:04.470942 1903 catalog_manager.cc:5856] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b reported cstate change: term changed from 2 to 3, leader changed from 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67) to a157015bf640485fb805159465a9ab1b (127.0.96.66). New cstate: current_term: 3 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:04.487273 2259 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:04.530175 2199 tablet_service.cc:2056] Received LeaderStepDown RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a"
dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
from {username='slave'} at 127.0.0.1:52878
I20261120 03:03:04.530809 2199 raft_consensus.cc:635] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Received request to step down
I20261120 03:03:04.531615 2199 raft_consensus.cc:3291] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Stepping down as leader of term 2
I20261120 03:03:04.531930 2199 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:04.532711 2199 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.534883 2199 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:04.680572 2323 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:04.681022 2323 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.682547 2323 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:04.683568 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:04.683581 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:04.684376 2199 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 3
I20261120 03:03:04.685389 1994 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:04.686075 2323 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261120 03:03:04.686462 2323 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:04.686719 2323 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:04.687275 2323 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.692167 2323 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 election: Requested vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:04.692848 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:04.693065 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:04.694150 2199 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 3
I20261120 03:03:04.695854 2259 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:04.696345 2259 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.698661 2259 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:04.699772 2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:04.700182 1994 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:04.700533 2198 raft_consensus.cc:2628] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a157015bf640485fb805159465a9ab1b in current term 3: Already voted for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in this term.
I20261120 03:03:04.700992 2323 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Leader election won for term 3
I20261120 03:03:04.701447 2323 raft_consensus.cc:744] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 LEADER]: Becoming Leader. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:04.701792 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:04.702189 2323 consensus_queue.cc:238] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.703548 2086 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:04.704597 2259 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:04.705399 2259 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20261120 03:03:04.711639 2198 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261120 03:03:04.711895 2120 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261120 03:03:04.714844 2256 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:04.716208 2256 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:04.716796 1905 catalog_manager.cc:5856] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 2 to 3, leader changed from 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67) to 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65). New cstate: current_term: 3 leader_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:04.745097 1902 catalog_manager.cc:3588] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53956:
table { table_id: "267fd8b91bac4f058896b2ba899401ba" table_name: "test-table" } new_table_name: "NewTableName"
I20261120 03:03:04.775458 1904 catalog_manager.cc:5956] TS 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485) does not have the latest schema for tablet ff883ba4b464463a9628eff440f1190a (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba]). Expected version 1 got 0
I20261120 03:03:04.778101 1903 catalog_manager.cc:5956] TS 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169) does not have the latest schema for tablet ff883ba4b464463a9628eff440f1190a (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba]). Expected version 1 got 0
I20261120 03:03:04.780187 1905 catalog_manager.cc:5956] TS a157015bf640485fb805159465a9ab1b (127.0.96.66:36367) does not have the latest schema for tablet ff883ba4b464463a9628eff440f1190a (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba]). Expected version 1 got 0
I20261120 03:03:04.795710 385 master.cc:585] Master@127.0.96.124:46415 shutting down...
I20261120 03:03:04.829411 2336 tablet.cc:1872] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261120 03:03:04.844023 2339 tablet.cc:1872] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261120 03:03:04.883100 2340 tablet.cc:1872] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261120 03:03:04.879980 2343 tablet.cc:1872] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261120 03:03:04.907737 2341 tablet.cc:1872] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261120 03:03:04.892259 2338 tablet.cc:1872] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261120 03:03:04.919839 2344 tablet.cc:1872] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261120 03:03:04.914868 2342 tablet.cc:1872] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261120 03:03:04.952399 2339 tablet.cc:1872] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261120 03:03:05.002301 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 LEADER]: Raft consensus shutting down.
I20261120 03:03:05.002985 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:05.003314 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: stopping tablet replica
I20261120 03:03:05.024287 385 master.cc:607] Master@127.0.96.124:46415 shutdown complete.
I20261120 03:03:05.192318 2349 raft_consensus.cc:534] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a4598f24f6246008d66f45cff9355af)
I20261120 03:03:05.192816 2349 raft_consensus.cc:556] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.194777 2349 leader_election.cc:290] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:05.195574 1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:05.196460 1784 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate e68c557e0a6047f5956737cfd33fffb5 in term 2
I20261120 03:03:05.196489 2350 raft_consensus.cc:534] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a4598f24f6246008d66f45cff9355af)
I20261120 03:03:05.197257 2350 raft_consensus.cc:556] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.199270 2350 leader_election.cc:290] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:05.200058 1823 leader_election.cc:304] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 764428dd82184375a0c80cbf46397e89, e68c557e0a6047f5956737cfd33fffb5; no voters:
I20261120 03:03:05.202425 2349 raft_consensus.cc:3024] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261120 03:03:05.202809 2349 raft_consensus.cc:534] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3a4598f24f6246008d66f45cff9355af)
I20261120 03:03:05.203161 2349 raft_consensus.cc:3296] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:05.208755 2349 raft_consensus.cc:556] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.213550 1849 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5" is_pre_election: true
I20261120 03:03:05.214264 1849 raft_consensus.cc:2628] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 764428dd82184375a0c80cbf46397e89 in current term 3: Already voted for candidate e68c557e0a6047f5956737cfd33fffb5 in this term.
W20261120 03:03:05.218288 1825 leader_election.cc:336] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415): Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
I20261120 03:03:05.219265 2349 leader_election.cc:290] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 election: Requested vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:05.219856 1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89"
I20261120 03:03:05.220466 1784 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:05.221333 1784 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Leader election vote request: granting yes vote for candidate e68c557e0a6047f5956737cfd33fffb5 in term 3
W20261120 03:03:05.223701 1825 leader_election.cc:336] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415): Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
W20261120 03:03:05.223861 1760 leader_election.cc:336] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415): Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
I20261120 03:03:05.224325 1760 leader_election.cc:304] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 764428dd82184375a0c80cbf46397e89; no voters: 3a4598f24f6246008d66f45cff9355af, e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:05.225818 2356 raft_consensus.cc:2966] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20261120 03:03:05.227296 1823 leader_election.cc:304] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 764428dd82184375a0c80cbf46397e89, e68c557e0a6047f5956737cfd33fffb5; no voters: 3a4598f24f6246008d66f45cff9355af
I20261120 03:03:05.227952 2349 raft_consensus.cc:3024] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Leader election won for term 3
I20261120 03:03:05.228368 2349 raft_consensus.cc:744] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 LEADER]: Becoming Leader. State: Replica: e68c557e0a6047f5956737cfd33fffb5, State: Running, Role: LEADER
I20261120 03:03:05.228884 2349 consensus_queue.cc:238] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.232235 2353 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68c557e0a6047f5956737cfd33fffb5. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.232895 2353 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:05.234510 2359 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:05.234413 1784 raft_consensus.cc:1338] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Refusing update from remote peer e68c557e0a6047f5956737cfd33fffb5: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20261120 03:03:05.236523 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:05.236982 1825 consensus_peers.cc:597] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 -> Peer 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415): Couldn't send request to peer 3a4598f24f6246008d66f45cff9355af. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:05.239907 2359 catalog_manager.cc:694] Loaded metadata for table NewTableName [id=267fd8b91bac4f058896b2ba899401ba]
W20261120 03:03:05.241842 2360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:05.242283 2361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:05.241552 2353 consensus_queue.cc:1049] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20261120 03:03:05.246299 2350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68c557e0a6047f5956737cfd33fffb5. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
W20261120 03:03:05.246925 2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:05.246986 2350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.247373 2359 tablet_loader.cc:96] loaded metadata for tablet 178ee4361c534583b29d93a1e67d3c4f (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:05.248396 385 server_base.cc:1061] running on GCE node
I20261120 03:03:05.249442 2359 tablet_loader.cc:96] loaded metadata for tablet 791003cb42cf4e50a7666946f7ffad6b (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:05.249563 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:05.249975 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:05.250192 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143785250175 us; error 0 us; skew 500 ppm
I20261120 03:03:05.250988 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:05.250957 2359 tablet_loader.cc:96] loaded metadata for tablet ff883ba4b464463a9628eff440f1190a (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:05.252657 2359 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:05.252349 2357 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.254088 2350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.255016 2350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.254838 2357 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:05.256734 385 webserver.cc:577] Webserver started at http://127.0.96.124:36403/ using document root <none> and password file <none>
I20261120 03:03:05.257052 2359 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:05.257313 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:05.257325 2359 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:05.257791 385 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261120 03:03:05.261314 2359 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:05.261370 385 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20261120 03:03:05.265607 2368 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:05.265610 2359 catalog_manager.cc:6248] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: Loaded TSK: 0
I20261120 03:03:05.266644 385 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20261120 03:03:05.266649 2359 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:05.267189 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root
uuid: "3a4598f24f6246008d66f45cff9355af"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:05.267519 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:05.295306 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:05.296468 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:05.339562 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.124:46415
I20261120 03:03:05.339727 2419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.124:46415 every 8 connection(s)
I20261120 03:03:05.347899 2420 sys_catalog.cc:263] Verifying existing consensus state
I20261120 03:03:05.353943 2420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: Bootstrap starting.
I20261120 03:03:05.387678 2224 heartbeater.cc:345] Connected to a master server at 127.0.96.124:46415
I20261120 03:03:05.408479 2225 heartbeater.cc:500] Master 127.0.96.125:41159 was elected leader, sending a full tablet report...
I20261120 03:03:05.418229 2147 heartbeater.cc:500] Master 127.0.96.125:41159 was elected leader, sending a full tablet report...
I20261120 03:03:05.421371 2146 heartbeater.cc:345] Connected to a master server at 127.0.96.124:46415
I20261120 03:03:05.422891 2069 heartbeater.cc:500] Master 127.0.96.125:41159 was elected leader, sending a full tablet report...
I20261120 03:03:05.437090 2420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=7 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:05.438274 2420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: Bootstrap complete.
I20261120 03:03:05.439838 2068 heartbeater.cc:345] Connected to a master server at 127.0.96.124:46415
I20261120 03:03:05.441468 2420 raft_consensus.cc:366] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.442163 2420 raft_consensus.cc:787] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a4598f24f6246008d66f45cff9355af, State: Initialized, Role: FOLLOWER
I20261120 03:03:05.442874 2420 consensus_queue.cc:261] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.446884 2420 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:05.447347 2395 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:05.445328 2425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.450026 2425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.452018 1837 catalog_manager.cc:6409] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: NewTableName [id=267fd8b91bac4f058896b2ba899401ba] alter complete (version 1)
I20261120 03:03:05.476099 2425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.476888 2425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.477728 2428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68c557e0a6047f5956737cfd33fffb5. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.478266 2428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.480194 2358 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.480944 2358 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:05.481473 2441 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:05.481712 2441 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: loading cluster ID for follower catalog manager: success
I20261120 03:03:05.486949 2441 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: acquiring CA information for follower catalog manager: success
I20261120 03:03:05.490151 2441 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261120 03:03:05.499287 2385 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" instance_seqno: 1795143781966557) as {username='slave'} at 127.0.0.1:57132; Asking this server to re-register.
I20261120 03:03:05.500416 2224 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:05.502899 2385 ts_manager.cc:210] Registered new tserver with Master: 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:05.531368 2385 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a157015bf640485fb805159465a9ab1b" instance_seqno: 1795143781803959) as {username='slave'} at 127.0.0.1:57140; Asking this server to re-register.
I20261120 03:03:05.532611 2146 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:05.535056 2385 ts_manager.cc:210] Registered new tserver with Master: a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:05.548032 2385 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" instance_seqno: 1795143781625822) as {username='slave'} at 127.0.0.1:57154; Asking this server to re-register.
I20261120 03:03:05.549283 2068 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:05.552209 2385 ts_manager.cc:210] Registered new tserver with Master: 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:05.552872 385 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.2104213s
I20261120 03:03:05.567138 385 master.cc:585] Master@127.0.96.125:41159 shutting down...
I20261120 03:03:05.593888 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 LEADER]: Raft consensus shutting down.
I20261120 03:03:05.594985 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:05.595468 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: stopping tablet replica
I20261120 03:03:05.636190 385 master.cc:607] Master@127.0.96.125:41159 shutdown complete.
I20261120 03:03:05.682816 2428 raft_consensus.cc:534] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e68c557e0a6047f5956737cfd33fffb5)
I20261120 03:03:05.683332 2428 raft_consensus.cc:556] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.685425 2428 leader_election.cc:290] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159)
W20261120 03:03:05.691433 2372 leader_election.cc:336] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159): Network error: Client connection negotiation failed: client connection to 127.0.96.125:41159: connect: Connection refused (error 111)
I20261120 03:03:05.695572 1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:05.696635 2369 leader_election.cc:304] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af; no voters: 764428dd82184375a0c80cbf46397e89, e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:05.697260 2428 raft_consensus.cc:2966] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:05.774529 2447 raft_consensus.cc:534] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e68c557e0a6047f5956737cfd33fffb5)
I20261120 03:03:05.774981 2447 raft_consensus.cc:556] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.776648 2447 leader_election.cc:290] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
W20261120 03:03:05.793767 1761 leader_election.cc:336] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159): Network error: Client connection negotiation failed: client connection to 127.0.96.125:41159: connect: Connection refused (error 111)
I20261120 03:03:05.795902 2395 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:05.796558 2395 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 764428dd82184375a0c80cbf46397e89 in term 3
I20261120 03:03:05.797650 1760 leader_election.cc:304] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89; no voters: e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:05.798246 2447 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261120 03:03:05.798602 2447 raft_consensus.cc:534] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e68c557e0a6047f5956737cfd33fffb5)
I20261120 03:03:05.798875 2447 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:05.800058 2447 raft_consensus.cc:556] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.805353 2447 leader_election.cc:290] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 4 election: Requested vote from peers e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:05.806331 2395 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af"
I20261120 03:03:05.806816 2395 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:05.807538 2395 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Leader election vote request: granting yes vote for candidate 764428dd82184375a0c80cbf46397e89 in term 4
W20261120 03:03:05.808777 1761 leader_election.cc:336] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159): Network error: Client connection negotiation failed: client connection to 127.0.96.125:41159: connect: Connection refused (error 111)
I20261120 03:03:05.812390 1760 leader_election.cc:304] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89; no voters: e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:05.813007 2447 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 4 FOLLOWER]: Leader election won for term 4
I20261120 03:03:05.813411 2447 raft_consensus.cc:744] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 4 LEADER]: Becoming Leader. State: Replica: 764428dd82184375a0c80cbf46397e89, State: Running, Role: LEADER
I20261120 03:03:05.814069 2447 consensus_queue.cc:238] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.816617 2451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764428dd82184375a0c80cbf46397e89. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.817345 2451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:05.819267 2395 raft_consensus.cc:1338] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Refusing update from remote peer 764428dd82184375a0c80cbf46397e89: Log matching property violated. Preceding OpId in replica: term: 3 index: 18. Preceding OpId from leader: term: 4 index: 19. (index mismatch)
I20261120 03:03:05.820189 2454 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:05.820703 2451 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20261120 03:03:05.822624 1761 consensus_peers.cc:597] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 -> Peer e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159): Couldn't send request to peer e68c557e0a6047f5956737cfd33fffb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.125:41159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:05.823828 2428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764428dd82184375a0c80cbf46397e89. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.824414 2428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.825618 2454 catalog_manager.cc:694] Loaded metadata for table NewTableName [id=267fd8b91bac4f058896b2ba899401ba]
I20261120 03:03:05.826213 2447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.826894 2447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:05.828104 2428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.828778 2428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.830032 2454 tablet_loader.cc:96] loaded metadata for tablet 178ee4361c534583b29d93a1e67d3c4f (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:05.831029 2454 tablet_loader.cc:96] loaded metadata for tablet 791003cb42cf4e50a7666946f7ffad6b (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:05.831966 2454 tablet_loader.cc:96] loaded metadata for tablet ff883ba4b464463a9628eff440f1190a (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:05.832973 2454 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:05.835767 2454 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:05.836005 2454 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:05.839247 2454 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:05.842079 2454 catalog_manager.cc:6248] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: Loaded TSK: 0
I20261120 03:03:05.842758 2454 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:05.848979 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:05.853612 2455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:05.854753 2456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:05.856671 2458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:05.857261 385 server_base.cc:1061] running on GCE node
I20261120 03:03:05.858091 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:05.858291 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:05.858445 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143785858435 us; error 0 us; skew 500 ppm
I20261120 03:03:05.858842 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:05.860872 385 webserver.cc:577] Webserver started at http://127.0.96.125:43061/ using document root <none> and password file <none>
I20261120 03:03:05.861285 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:05.861425 385 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261120 03:03:05.865442 385 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20261120 03:03:05.868069 2463 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:05.868852 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 03:03:05.869194 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root
uuid: "e68c557e0a6047f5956737cfd33fffb5"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:05.869518 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:05.879878 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:05.880748 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:05.923377 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.125:41159
I20261120 03:03:05.923481 2514 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.125:41159 every 8 connection(s)
I20261120 03:03:05.931491 2515 sys_catalog.cc:263] Verifying existing consensus state
I20261120 03:03:05.935626 2515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: Bootstrap starting.
I20261120 03:03:05.975961 2070 heartbeater.cc:500] Master 127.0.96.126:43725 was elected leader, sending a full tablet report...
I20261120 03:03:05.992908 2148 heartbeater.cc:500] Master 127.0.96.126:43725 was elected leader, sending a full tablet report...
I20261120 03:03:05.993736 2226 heartbeater.cc:500] Master 127.0.96.126:43725 was elected leader, sending a full tablet report...
I20261120 03:03:05.999846 2515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=7 ignored=0} mutations{seen=14 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:06.000887 2515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: Bootstrap complete.
I20261120 03:03:06.003957 2515 raft_consensus.cc:366] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:06.004693 2515 raft_consensus.cc:787] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68c557e0a6047f5956737cfd33fffb5, State: Initialized, Role: FOLLOWER
I20261120 03:03:06.005337 2515 consensus_queue.cc:261] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:06.013132 2522 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:06.013856 2522 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:06.014706 2515 sys_catalog.cc:565] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:06.019256 2490 raft_consensus.cc:3296] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:06.025790 2069 heartbeater.cc:345] Connected to a master server at 127.0.96.125:41159
I20261120 03:03:06.033200 2225 heartbeater.cc:345] Connected to a master server at 127.0.96.125:41159
I20261120 03:03:06.036993 2147 heartbeater.cc:345] Connected to a master server at 127.0.96.125:41159
I20261120 03:03:06.048633 2527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:06.049407 2527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:06.049805 2522 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764428dd82184375a0c80cbf46397e89. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:06.050613 2522 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:06.055310 2536 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:06.055588 2536 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: loading cluster ID for follower catalog manager: success
I20261120 03:03:06.060144 2536 catalog_manager.cc:1625] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: acquiring CA information for follower catalog manager: success
I20261120 03:03:06.063299 2536 catalog_manager.cc:1653] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261120 03:03:06.111029 2453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:06.111796 2453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:06.134670 2480 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" instance_seqno: 1795143781625822) as {username='slave'} at 127.0.0.1:37252; Asking this server to re-register.
I20261120 03:03:06.135811 2069 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:06.138926 2480 ts_manager.cc:210] Registered new tserver with Master: 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:06.140609 2479 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" instance_seqno: 1795143781966557) as {username='slave'} at 127.0.0.1:37246; Asking this server to re-register.
I20261120 03:03:06.142907 2225 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:06.144814 2479 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a157015bf640485fb805159465a9ab1b" instance_seqno: 1795143781803959) as {username='slave'} at 127.0.0.1:37268; Asking this server to re-register.
I20261120 03:03:06.145712 2480 ts_manager.cc:210] Registered new tserver with Master: 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:06.145875 2147 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:06.148909 2480 ts_manager.cc:210] Registered new tserver with Master: a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:06.148983 385 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.222978453s
I20261120 03:03:08.421607 2521 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:08.422322 2521 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:08.425918 2521 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:08.437839 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:08.489126 2541 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:08.489840 2541 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:08.493299 2541 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
W20261120 03:03:08.496539 2086 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.96.65:40485, user_credentials={real_user=slave}} blocked reactor thread for 59894.2us
I20261120 03:03:08.499097 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:08.513994 2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:08.514659 2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:08.516829 2543 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:08.517311 2543 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:08.517767 2085 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:08.518551 2542 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:08.519375 2085 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:08.519688 2543 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:08.520341 2542 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:08.522245 2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:08.533748 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:08.535211 2164 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:08.536406 2543 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:08.571172 2537 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:08.571661 2537 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:08.573315 2537 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:08.573787 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:08.574211 2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:08.575402 1994 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b
I20261120 03:03:08.576069 2537 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20261120 03:03:09.427834 2064 debug-util.cc:398] Leaking SignalData structure 0x7b08000fe7c0 after lost signal to thread 388
W20261120 03:03:09.429066 2064 debug-util.cc:398] Leaking SignalData structure 0x7b0800125560 after lost signal to thread 1808
W20261120 03:03:09.430126 2064 debug-util.cc:398] Leaking SignalData structure 0x7b08001190c0 after lost signal to thread 2067
W20261120 03:03:09.431149 2064 debug-util.cc:398] Leaking SignalData structure 0x7b08000c9220 after lost signal to thread 2145
W20261120 03:03:09.432082 2064 debug-util.cc:398] Leaking SignalData structure 0x7b080017a0c0 after lost signal to thread 2223
W20261120 03:03:09.433106 2064 debug-util.cc:398] Leaking SignalData structure 0x7b08000306a0 after lost signal to thread 2419
W20261120 03:03:09.434125 2064 debug-util.cc:398] Leaking SignalData structure 0x7b080002e8e0 after lost signal to thread 2514
I20261120 03:03:25.919606 2570 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:25.920817 2570 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:25.966727 2570 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:25.985739 2566 raft_consensus.cc:534] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:25.986351 2566 raft_consensus.cc:556] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:25.993346 2566 leader_election.cc:290] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159)
I20261120 03:03:25.990624 2567 raft_consensus.cc:534] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:25.995062 2567 raft_consensus.cc:556] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:25.998967 2567 leader_election.cc:290] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:26.001551 2572 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:26.002071 2572 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.004706 2573 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:26.005141 2573 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.005653 2572 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:26.007097 2573 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:26.024778 2576 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:26.025405 2576 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:26.027194 2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:26.028138 2198 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 3
I20261120 03:03:26.028476 2576 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:26.029687 1994 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters:
I20261120 03:03:26.031165 2570 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261120 03:03:26.031587 2570 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:26.031955 2570 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Advancing to term 4
W20261120 03:03:26.032454 2467 thread.cc:642] raft [worker] (thread pool) Time spent creating pthread: real 0.980s user 0.000s sys 0.016s
I20261120 03:03:26.032873 2570 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
W20261120 03:03:26.034677 2467 thread.cc:609] raft [worker] (thread pool) Time spent starting thread: real 0.983s user 0.000s sys 0.017s
I20261120 03:03:26.038396 2576 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:26.038846 2576 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.043550 2576 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:26.047367 2570 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 4 election: Requested vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:26.049599 2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:26.050231 2198 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:26.051043 2198 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 4
I20261120 03:03:26.059181 1994 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters:
I20261120 03:03:26.059965 2570 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader election won for term 4
I20261120 03:03:26.060451 2570 raft_consensus.cc:744] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 LEADER]: Becoming Leader. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:26.061314 2570 consensus_queue.cc:238] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.064882 2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:26.065409 2198 raft_consensus.cc:1303] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Rejecting Update request from peer a157015bf640485fb805159465a9ab1b for earlier term 3. Current term is 4. Ops: []
I20261120 03:03:26.065567 2121 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 3
I20261120 03:03:26.066212 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:26.066807 2576 consensus_queue.cc:1060] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261120 03:03:26.067619 2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:26.067827 2576 raft_consensus.cc:3291] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 3 LEADER]: Stepping down as leader of term 3
I20261120 03:03:26.069126 2118 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:26.071591 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:26.068394 2576 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:26.074138 2117 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:26.078526 2042 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:26.079425 2041 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:26.081430 2576 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.086398 2040 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:26.090018 2043 raft_consensus.cc:1303] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 LEADER]: Rejecting Update request from peer a157015bf640485fb805159465a9ab1b for earlier term 3. Current term is 4. Ops: []
I20261120 03:03:26.089625 2576 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:26.094553 2086 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:26.095289 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:26.095645 2086 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:26.099583 2574 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:26.100436 2574 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:26.101238 2165 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters:
I20261120 03:03:26.103813 2573 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261120 03:03:26.104112 2573 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:26.104579 2573 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:26.105736 2573 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.109328 2198 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20261120 03:03:26.114035 2371 thread.cc:642] raft [worker] (thread pool) Time spent creating pthread: real 1.078s user 0.004s sys 0.025s
W20261120 03:03:26.114409 2371 thread.cc:609] raft [worker] (thread pool) Time spent starting thread: real 1.079s user 0.004s sys 0.025s
I20261120 03:03:26.116346 2573 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:26.122375 2577 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20261120 03:03:26.124555 2041 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:26.135413 1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:26.153277 2165 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:26.154888 2119 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:26.156370 2573 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:26.158819 2165 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:26.159762 2573 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20261120 03:03:26.162674 2197 raft_consensus.cc:1303] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Rejecting Update request from peer 9ac1cc0b02fd4724b4680da6a7e2a875 for earlier term 3. Current term is 4. Ops: []
I20261120 03:03:26.139104 1771 catalog_manager.cc:5856] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 3 to 4, leader changed from a157015bf640485fb805159465a9ab1b (127.0.96.66) to 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65). New cstate: current_term: 4 leader_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:26.164039 2577 consensus_queue.cc:1060] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261120 03:03:26.164544 2117 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 4
I20261120 03:03:26.164916 2577 raft_consensus.cc:3291] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 LEADER]: Stepping down as leader of term 3
I20261120 03:03:26.165241 2577 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:26.165911 1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:26.165952 2120 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20261120 03:03:26.165961 2577 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.170043 2577 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:26.172508 2395 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:26.174294 2490 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5" is_pre_election: true
I20261120 03:03:26.174866 2466 leader_election.cc:304] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68c557e0a6047f5956737cfd33fffb5; no voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89
I20261120 03:03:26.176299 2567 raft_consensus.cc:2966] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261120 03:03:26.178485 2372 leader_election.cc:304] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af; no voters: 764428dd82184375a0c80cbf46397e89, e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:26.179473 2566 raft_consensus.cc:2966] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261120 03:03:26.183800 2568 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20261120 03:03:26.351231 2582 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:26.351682 2582 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.353302 2582 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:26.353926 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:26.354166 2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:26.354449 2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 4
I20261120 03:03:26.354815 2197 raft_consensus.cc:2628] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a157015bf640485fb805159465a9ab1b in current term 4: Already voted for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in this term.
I20261120 03:03:26.355242 2086 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters:
I20261120 03:03:26.355885 2582 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261120 03:03:26.356186 2582 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:26.356427 2582 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:26.357017 2582 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.362284 2582 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:26.362972 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:26.363145 2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:26.363799 2197 raft_consensus.cc:2628] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a157015bf640485fb805159465a9ab1b in current term 4: Already voted for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in this term.
I20261120 03:03:26.363834 2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 4
I20261120 03:03:26.371343 2086 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c
I20261120 03:03:26.372053 2582 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader election won for term 4
I20261120 03:03:26.372620 2582 raft_consensus.cc:744] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 4 LEADER]: Becoming Leader. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:26.373708 2582 consensus_queue.cc:238] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.379702 2197 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Refusing update from remote peer a157015bf640485fb805159465a9ab1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20261120 03:03:26.380620 2043 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Refusing update from remote peer a157015bf640485fb805159465a9ab1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20261120 03:03:26.381104 2582 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261120 03:03:26.382413 2582 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261120 03:03:26.397804 1771 catalog_manager.cc:5856] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b reported cstate change: term changed from 3 to 4, leader changed from 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65) to a157015bf640485fb805159465a9ab1b (127.0.96.66). New cstate: current_term: 4 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } }
W20261120 03:03:26.966791 390 debug-util.cc:398] Leaking SignalData structure 0x7b080002c2a0 after lost signal to thread 2371
W20261120 03:03:27.104789 2511 debug-util.cc:398] Leaking SignalData structure 0x7b08000e41c0 after lost signal to thread 388
W20261120 03:03:27.105719 2511 debug-util.cc:398] Leaking SignalData structure 0x7b0800172b60 after lost signal to thread 1808
W20261120 03:03:27.106745 2511 debug-util.cc:398] Leaking SignalData structure 0x7b08001809c0 after lost signal to thread 2067
W20261120 03:03:27.107558 2511 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5fa0 after lost signal to thread 2145
W20261120 03:03:27.108366 2511 debug-util.cc:398] Leaking SignalData structure 0x7b0800052220 after lost signal to thread 2223
W20261120 03:03:27.109175 2511 debug-util.cc:398] Leaking SignalData structure 0x7b080016adc0 after lost signal to thread 2419
W20261120 03:03:27.109957 2511 debug-util.cc:398] Leaking SignalData structure 0x7b0800059280 after lost signal to thread 2514
W20261120 03:03:29.078586 2064 debug-util.cc:398] Leaking SignalData structure 0x7b080002eb40 after lost signal to thread 388
I20261120 03:03:29.170542 2608 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:29.171120 2608 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
W20261120 03:03:29.172451 1994 thread.cc:642] raft [worker] (thread pool) Time spent creating pthread: real 0.918s user 0.011s sys 0.012s
W20261120 03:03:29.172875 1994 thread.cc:609] raft [worker] (thread pool) Time spent starting thread: real 0.919s user 0.011s sys 0.012s
I20261120 03:03:29.176059 2608 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:29.215683 2479 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37246 (request call id 206) took 1156 ms. Trace:
I20261120 03:03:29.216790 2479 rpcz_store.cc:276] 1120 03:03:28.058935 (+ 0us) service_pool.cc:167] Inserting onto call queue
1120 03:03:28.075291 (+ 16356us) service_pool.cc:224] Handling call
1120 03:03:29.215262 (+1139971us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20261120 03:03:29.470950 1994 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.96.67:35169 timed out after 0.263s (SENT) [suppressed 18 similar messages]
W20261120 03:03:29.763438 1994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.67:35169, user_credentials={real_user=slave}} blocked reactor thread for 292647us
I20261120 03:03:29.954772 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
W20261120 03:03:30.067462 1994 leader_election.cc:336] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169): Timed out: RequestConsensusVote RPC to 127.0.96.67:35169 timed out after 0.263s (SENT)
I20261120 03:03:30.082844 2199 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 4
W20261120 03:03:30.083700 2199 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52464 (request call id 487) took 279 ms (client timeout 263 ms). Trace:
W20261120 03:03:30.084311 2199 rpcz_store.cc:269] 1120 03:03:29.804667 (+ 0us) service_pool.cc:167] Inserting onto call queue
1120 03:03:29.954020 (+149353us) service_pool.cc:224] Handling call
1120 03:03:30.083549 (+129529us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20261120 03:03:30.644570 2611 raft_consensus.cc:534] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:30.645686 2611 raft_consensus.cc:556] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:30.649366 2611 leader_election.cc:290] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159)
I20261120 03:03:30.650527 1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:30.667115 2613 raft_consensus.cc:534] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:30.667702 2613 raft_consensus.cc:556] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:30.670565 2613 leader_election.cc:290] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:30.685091 2614 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:30.685746 2614 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:30.688694 2614 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:30.974197 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:30.975368 2199 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 4
W20261120 03:03:31.069312 2466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.124:46415, user_credentials={real_user=slave}} blocked reactor thread for 144242us
W20261120 03:03:31.070713 2085 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.67:35169, user_credentials={real_user=slave}} blocked reactor thread for 116407us
I20261120 03:03:31.074254 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:31.075187 2199 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 4
I20261120 03:03:31.137580 2197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52464 (request call id 486) took 1909 ms. Trace:
I20261120 03:03:31.138012 2197 rpcz_store.cc:276] 1120 03:03:29.228230 (+ 0us) service_pool.cc:167] Inserting onto call queue
1120 03:03:31.136437 (+1908207us) service_pool.cc:224] Handling call
1120 03:03:31.136815 (+ 378us) raft_consensus.cc:1536] Updating replica for 0 ops
1120 03:03:31.137062 (+ 247us) raft_consensus.cc:1602] Early marking committed up to index 0
1120 03:03:31.137111 (+ 49us) raft_consensus.cc:1607] Triggering prepare for 0 ops
1120 03:03:31.137164 (+ 53us) raft_consensus.cc:1713] Marking committed up to 6
1120 03:03:31.137201 (+ 37us) raft_consensus.cc:1765] Filling consensus response to leader.
1120 03:03:31.137279 (+ 78us) raft_consensus.cc:1759] UpdateReplicas() finished
1120 03:03:31.137433 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20261120 03:03:31.139463 1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:31.142298 2085 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters:
W20261120 03:03:31.142944 2198 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52464 (request call id 485) took 1915 ms (client timeout 263 ms). Trace:
I20261120 03:03:31.145078 2614 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
W20261120 03:03:31.143386 2198 rpcz_store.cc:269] 1120 03:03:29.227776 (+ 0us) service_pool.cc:167] Inserting onto call queue
1120 03:03:31.142633 (+1914857us) service_pool.cc:211] Skipping call since client already timed out
1120 03:03:31.142758 (+ 125us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20261120 03:03:31.146209 2614 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:31.148711 2618 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:31.152758 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:31.150465 2618 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.154177 2199 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 3
I20261120 03:03:31.156332 2085 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters:
W20261120 03:03:31.157568 1996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.66:36367, user_credentials={real_user=slave}} blocked reactor thread for 225186us
I20261120 03:03:31.160411 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.161027 2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 4
I20261120 03:03:31.162818 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.164433 2164 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters:
I20261120 03:03:31.156929 2618 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:31.168243 2619 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20261120 03:03:31.170053 2118 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.171126 2614 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:31.174387 1996 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b
W20261120 03:03:31.175002 2064 debug-util.cc:398] Leaking SignalData structure 0x7b0800173d60 after lost signal to thread 1808
I20261120 03:03:31.167487 2618 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.176203 2618 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
W20261120 03:03:31.176750 2064 debug-util.cc:398] Leaking SignalData structure 0x7b080012aee0 after lost signal to thread 2067
I20261120 03:03:31.177482 2620 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20261120 03:03:31.177868 2620 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting leader election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:31.178220 2618 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:31.178995 2614 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261120 03:03:31.179212 2620 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Advancing to term 5
I20261120 03:03:31.179543 2614 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.180032 2614 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:31.171741 2619 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.183588 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.182703 2619 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Advancing to term 5
I20261120 03:03:31.180931 2614 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:31.189023 2118 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.189772 2584 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261120 03:03:31.190753 2117 raft_consensus.cc:1303] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Rejecting Update request from peer 9ac1cc0b02fd4724b4680da6a7e2a875 for earlier term 3. Current term is 4. Ops: []
I20261120 03:03:31.196256 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.197242 2395 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:31.205560 2490 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5" is_pre_election: true
I20261120 03:03:31.206918 2394 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:31.209281 2372 leader_election.cc:304] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af; no voters: 764428dd82184375a0c80cbf46397e89, e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:31.210539 2466 leader_election.cc:304] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68c557e0a6047f5956737cfd33fffb5; no voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89
I20261120 03:03:31.204623 2620 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
W20261120 03:03:31.218573 2064 debug-util.cc:398] Leaking SignalData structure 0x7b08000a04e0 after lost signal to thread 2145
I20261120 03:03:31.222122 2620 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
W20261120 03:03:31.222885 2164 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.65:40485, user_credentials={real_user=slave}} blocked reactor thread for 58607.2us
I20261120 03:03:31.224704 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.226413 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:31.226778 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:31.227283 2043 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Advancing to term 5
I20261120 03:03:31.228353 2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Leader election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 5
I20261120 03:03:31.229494 2164 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:31.236379 2620 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20261120 03:03:31.241268 2064 debug-util.cc:398] Leaking SignalData structure 0x7b0800239e40 after lost signal to thread 2223
W20261120 03:03:31.242628 2064 debug-util.cc:398] Leaking SignalData structure 0x7b080023b6a0 after lost signal to thread 2419
I20261120 03:03:31.246129 2607 consensus_queue.cc:1060] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20261120 03:03:31.248309 2064 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9000 after lost signal to thread 2514
I20261120 03:03:31.256822 2619 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
W20261120 03:03:31.268473 2372 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.125:41159, user_credentials={real_user=slave}} blocked reactor thread for 59392.1us
I20261120 03:03:31.271310 2626 raft_consensus.cc:2966] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261120 03:03:31.278391 2197 raft_consensus.cc:1303] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Rejecting Update request from peer a157015bf640485fb805159465a9ab1b for earlier term 4. Current term is 5. Ops: []
I20261120 03:03:31.280175 2624 consensus_queue.cc:1060] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20261120 03:03:31.282713 2614 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 election: Requested vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:31.283425 2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
W20261120 03:03:31.288208 2086 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.96.65:40485, user_credentials={real_user=slave}} blocked reactor thread for 58205.5us
I20261120 03:03:31.309310 2619 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 5 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:31.310484 2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:31.314849 2164 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:31.315654 2625 raft_consensus.cc:2966] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261120 03:03:31.315944 2623 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Leader election won for term 5
I20261120 03:03:31.316617 2623 raft_consensus.cc:744] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 LEADER]: Becoming Leader. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:31.318415 2623 consensus_queue.cc:238] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.320418 2584 raft_consensus.cc:3291] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 LEADER]: Stepping down as leader of term 3
I20261120 03:03:31.320755 2584 raft_consensus.cc:787] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:31.322912 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.321527 2584 consensus_queue.cc:261] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:31.324043 2041 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:31.324589 2040 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:31.327000 2117 raft_consensus.cc:1303] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Rejecting Update request from peer 9ac1cc0b02fd4724b4680da6a7e2a875 for earlier term 4. Current term is 5. Ops: []
I20261120 03:03:31.327385 2086 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:31.330685 2614 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20261120 03:03:31.332157 2621 raft_consensus.cc:3291] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 4 LEADER]: Stepping down as leader of term 4
I20261120 03:03:31.335189 2621 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:31.336385 2621 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.328559 2086 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:31.340649 2614 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20261120 03:03:31.337630 2627 consensus_queue.cc:1060] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20261120 03:03:31.334273 2584 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:31.342845 2628 raft_consensus.cc:3291] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 LEADER]: Stepping down as leader of term 4
I20261120 03:03:31.343246 2628 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:31.344440 2041 raft_consensus.cc:1303] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Rejecting Update request from peer a157015bf640485fb805159465a9ab1b for earlier term 4. Current term is 5. Ops: []
I20261120 03:03:31.344277 2628 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.345388 2628 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Advancing to term 5
I20261120 03:03:31.346640 2040 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20261120 03:03:31.338574 2621 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Advancing to term 5
I20261120 03:03:31.358151 2043 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 4
I20261120 03:03:31.358773 2620 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20261120 03:03:31.351503 1773 catalog_manager.cc:5856] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c reported cstate change: term changed from 4 to 5, leader changed from a157015bf640485fb805159465a9ab1b (127.0.96.66) to 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67). New cstate: current_term: 5 leader_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:31.364535 2117 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20261120 03:03:31.366616 2620 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20261120 03:03:31.428067 2623 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.428699 2623 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.447011 2623 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:31.450060 2040 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.454062 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.454815 2120 raft_consensus.cc:2628] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 5: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:31.456002 2040 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 5
I20261120 03:03:31.459084 2164 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:31.460421 2620 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20261120 03:03:31.460778 2620 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.461153 2620 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Advancing to term 5
I20261120 03:03:31.461959 2620 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.460036 2623 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.463570 2623 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:31.467921 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.467929 2040 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.468798 2040 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 4
I20261120 03:03:31.468806 2120 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 4: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:31.470304 2164 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters:
I20261120 03:03:31.470265 2623 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:31.483491 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:31.486826 2620 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:31.488404 2618 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261120 03:03:31.498382 2120 raft_consensus.cc:2628] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 5: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:31.494966 2618 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.501966 2618 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:31.503022 2618 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
W20261120 03:03:31.514716 2164 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.65:40485, user_credentials={real_user=slave}} blocked reactor thread for 44472.6us
I20261120 03:03:31.515975 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:31.517017 2043 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Leader election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 5
I20261120 03:03:31.517405 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:31.518385 2120 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 4: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:31.519582 2618 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:31.525252 2040 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:31.527350 2040 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 4
I20261120 03:03:31.529824 2164 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:31.532694 2620 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Leader election won for term 5
I20261120 03:03:31.533797 2620 raft_consensus.cc:744] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 LEADER]: Becoming Leader. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:31.542305 2164 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:31.543284 2618 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader election won for term 4
I20261120 03:03:31.543023 2620 consensus_queue.cc:238] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.543918 2618 raft_consensus.cc:744] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 LEADER]: Becoming Leader. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:31.545071 2618 consensus_queue.cc:238] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:31.554735 2120 raft_consensus.cc:1338] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20261120 03:03:31.556659 2643 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20261120 03:03:31.560480 2040 raft_consensus.cc:1338] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20261120 03:03:31.562053 2643 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261120 03:03:31.572944 2621 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:31.573529 2624 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:31.573433 2621 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:31.577041 2624 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.578545 2621 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:31.578908 2624 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:31.581111 2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.581811 2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:31.582156 2041 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.582167 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:31.583480 2086 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:31.581825 2043 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 5
I20261120 03:03:31.585072 2086 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters:
I20261120 03:03:31.590294 2043 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20261120 03:03:31.599601 2620 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20261120 03:03:31.599571 1771 catalog_manager.cc:5856] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c reported cstate change: term changed from 4 to 5, leader changed from 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65) to 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67). New cstate: current_term: 5 leader_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:31.586694 2624 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20261120 03:03:31.609215 2117 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20261120 03:03:31.609843 2624 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:31.610425 2624 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Advancing to term 6
I20261120 03:03:31.611833 2624 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.613445 1771 catalog_manager.cc:5856] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c reported cstate change: term changed from 3 to 4, leader changed from 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65) to 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67). New cstate: current_term: 4 leader_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:31.614897 2618 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20261120 03:03:31.618289 2117 raft_consensus.cc:1303] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Rejecting Update request from peer 852e6c6e1ef54e5ebde6ab6f55b82b9c for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20261120 03:03:31.626492 2621 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261120 03:03:31.630473 2618 consensus_queue.cc:1060] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20261120 03:03:31.658401 2624 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 6 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:31.659304 2643 raft_consensus.cc:3291] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 LEADER]: Stepping down as leader of term 5
I20261120 03:03:31.659648 2643 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:31.659953 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:31.660420 2643 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.661688 2040 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:31.663844 2643 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Advancing to term 6
I20261120 03:03:31.671542 2199 raft_consensus.cc:2044] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Leader election vote request: denying vote to candidate a157015bf640485fb805159465a9ab1b for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than candidate's term: 4 index: 6
I20261120 03:03:31.672935 2085 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:31.673687 2614 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20261120 03:03:31.803313 2628 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:31.803930 2628 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.805915 2628 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:31.806906 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:31.807582 2199 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 6
I20261120 03:03:31.808615 1994 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters:
I20261120 03:03:31.809295 2628 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20261120 03:03:31.809687 2628 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:31.810071 2628 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Advancing to term 6
I20261120 03:03:31.810030 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.811012 2120 raft_consensus.cc:2628] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in current term 6: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:31.811048 2628 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.826243 2628 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 6 election: Requested vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:31.827225 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:31.827701 2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:31.828267 2199 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 6
I20261120 03:03:31.828418 2120 raft_consensus.cc:2628] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in current term 6: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:31.835383 1994 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:31.836467 2607 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 6 FOLLOWER]: Leader election won for term 6
I20261120 03:03:31.837337 2607 raft_consensus.cc:744] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 6 LEADER]: Becoming Leader. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:31.838129 2607 consensus_queue.cc:238] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.843328 2199 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20261120 03:03:31.843664 2120 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20261120 03:03:31.844969 2628 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20261120 03:03:31.847951 1772 catalog_manager.cc:5856] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 5 to 6, leader changed from 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67) to 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65). New cstate: current_term: 6 leader_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:31.851580 2628 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:784: Failure
Expected equality of these values:
live_row_count
Which is: 67
table_info->GetMetrics()->live_row_count->value()
Which is: 0
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:768: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 784, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:896: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20261120 03:03:36.371076 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:36.399328 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:36.399921 385 tablet_replica.cc:333] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: stopping tablet replica
I20261120 03:03:36.400439 385 raft_consensus.cc:2440] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.400847 385 raft_consensus.cc:2469] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.403102 385 tablet_replica.cc:333] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: stopping tablet replica
W20261120 03:03:36.403208 2164 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20261120 03:03:36.403661 385 raft_consensus.cc:2440] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 6 LEADER]: Raft consensus shutting down.
I20261120 03:03:36.404613 385 raft_consensus.cc:2469] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 6 FOLLOWER]: Raft consensus is shut down!
W20261120 03:03:36.405910 2164 consensus_peers.cc:597] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c -> Peer 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485): Couldn't send request to peer 9ac1cc0b02fd4724b4680da6a7e2a875. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:36.406553 385 tablet_replica.cc:333] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: stopping tablet replica
I20261120 03:03:36.406987 385 raft_consensus.cc:2440] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.407342 385 raft_consensus.cc:2469] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Raft consensus is shut down!
W20261120 03:03:36.435228 2164 consensus_peers.cc:597] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c -> Peer 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485): Couldn't send request to peer 9ac1cc0b02fd4724b4680da6a7e2a875. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.65:40485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:36.461351 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:36.475200 385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:36.504944 2664 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:36.505401 2664 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:36.506110 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:36.506793 385 tablet_replica.cc:333] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: stopping tablet replica
I20261120 03:03:36.507069 2664 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:36.507611 385 raft_consensus.cc:2440] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.508332 385 raft_consensus.cc:2469] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.509191 2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:36.510725 385 tablet_replica.cc:333] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: stopping tablet replica
I20261120 03:03:36.511359 385 raft_consensus.cc:2440] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.511859 385 raft_consensus.cc:2469] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Raft consensus is shut down!
W20261120 03:03:36.512672 2086 leader_election.cc:336] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485): Network error: Client connection negotiation failed: client connection to 127.0.96.65:40485: connect: Connection refused (error 111)
I20261120 03:03:36.513085 2086 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
W20261120 03:03:36.513708 2086 raft_consensus.cc:2902] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20261120 03:03:36.514537 385 tablet_replica.cc:333] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: stopping tablet replica
I20261120 03:03:36.515241 385 raft_consensus.cc:2440] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.515856 385 raft_consensus.cc:2469] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Raft consensus is shut down!
W20261120 03:03:36.537401 2165 consensus_peers.cc:597] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c -> Peer a157015bf640485fb805159465a9ab1b (127.0.96.66:36367): Couldn't send request to peer a157015bf640485fb805159465a9ab1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.66:36367: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:36.546162 385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:36.559314 385 tablet_server.cc:179] TabletServer@127.0.96.67:0 shutting down...
W20261120 03:03:36.574368 2165 consensus_peers.cc:597] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c -> Peer a157015bf640485fb805159465a9ab1b (127.0.96.66:36367): Couldn't send request to peer a157015bf640485fb805159465a9ab1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.66:36367: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:36.585305 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:36.585899 385 tablet_replica.cc:333] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: stopping tablet replica
I20261120 03:03:36.586660 385 raft_consensus.cc:2440] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.587358 385 raft_consensus.cc:2469] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.589263 385 tablet_replica.cc:333] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: stopping tablet replica
I20261120 03:03:36.589651 385 raft_consensus.cc:2440] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 LEADER]: Raft consensus shutting down.
I20261120 03:03:36.590348 385 raft_consensus.cc:2469] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.592000 385 tablet_replica.cc:333] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: stopping tablet replica
I20261120 03:03:36.592484 385 raft_consensus.cc:2440] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 LEADER]: Raft consensus shutting down.
I20261120 03:03:36.593086 385 raft_consensus.cc:2469] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.614674 385 tablet_server.cc:196] TabletServer@127.0.96.67:0 shutdown complete.
I20261120 03:03:36.626992 385 master.cc:585] Master@127.0.96.126:43725 shutting down...
I20261120 03:03:36.647610 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 4 LEADER]: Raft consensus shutting down.
I20261120 03:03:36.648514 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.648893 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: stopping tablet replica
I20261120 03:03:36.669073 385 master.cc:607] Master@127.0.96.126:43725 shutdown complete.
I20261120 03:03:36.685151 385 master.cc:585] Master@127.0.96.125:41159 shutting down...
I20261120 03:03:36.704452 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.704883 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.705148 385 tablet_replica.cc:333] stopping tablet replica
I20261120 03:03:36.740841 385 master.cc:607] Master@127.0.96.125:41159 shutdown complete.
I20261120 03:03:36.753082 385 master.cc:585] Master@127.0.96.124:46415 shutting down...
I20261120 03:03:36.774307 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.774768 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.775199 385 tablet_replica.cc:333] stopping tablet replica
I20261120 03:03:36.793802 385 master.cc:607] Master@127.0.96.124:46415 shutdown complete.
I20261120 03:03:36.816525 385 test_util.cc:182] -----------------------------------------------
I20261120 03:03:36.816721 385 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0
[ FAILED ] TsTabletManagerITest.TestTableStats (35709 ms)
[ RUN ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20261120 03:03:36.822628 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:35193
I20261120 03:03:36.823696 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:36.829058 2674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:36.830194 2675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:36.833142 2677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:36.833853 385 server_base.cc:1061] running on GCE node
I20261120 03:03:36.835075 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:36.835322 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:36.835485 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143816835469 us; error 0 us; skew 500 ppm
I20261120 03:03:36.835983 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:36.838282 385 webserver.cc:577] Webserver started at http://127.0.96.126:44697/ using document root <none> and password file <none>
I20261120 03:03:36.838768 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:36.838964 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:36.839231 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:36.840350 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "48f97e77c6dc4c81886519ca702f6bc9"
format_stamp: "Formatted at 2026-11-20 03:03:36 on dist-test-slave-54pf"
I20261120 03:03:36.844494 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20261120 03:03:36.847424 2682 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:36.848256 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261120 03:03:36.848469 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "48f97e77c6dc4c81886519ca702f6bc9"
format_stamp: "Formatted at 2026-11-20 03:03:36 on dist-test-slave-54pf"
I20261120 03:03:36.848675 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:36.873592 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:36.874595 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:36.908597 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:35193
I20261120 03:03:36.908674 2733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:35193 every 8 connection(s)
I20261120 03:03:36.912262 2734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:36.921344 2734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9: Bootstrap starting.
I20261120 03:03:36.925904 2734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:36.929539 2734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9: No bootstrap required, opened a new log
I20261120 03:03:36.931535 2734 raft_consensus.cc:366] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f97e77c6dc4c81886519ca702f6bc9" member_type: VOTER }
I20261120 03:03:36.931910 2734 raft_consensus.cc:392] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:36.932147 2734 raft_consensus.cc:787] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f97e77c6dc4c81886519ca702f6bc9, State: Initialized, Role: FOLLOWER
I20261120 03:03:36.932753 2734 consensus_queue.cc:261] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f97e77c6dc4c81886519ca702f6bc9" member_type: VOTER }
I20261120 03:03:36.933507 2734 raft_consensus.cc:407] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:36.933748 2734 raft_consensus.cc:534] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:36.933959 2734 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:36.934943 2734 raft_consensus.cc:556] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f97e77c6dc4c81886519ca702f6bc9" member_type: VOTER }
I20261120 03:03:36.938930 2734 leader_election.cc:304] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f97e77c6dc4c81886519ca702f6bc9; no voters:
I20261120 03:03:36.939427 2734 leader_election.cc:290] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:36.939512 2737 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:36.940938 2737 raft_consensus.cc:744] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 48f97e77c6dc4c81886519ca702f6bc9, State: Running, Role: LEADER
I20261120 03:03:36.941531 2737 consensus_queue.cc:238] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f97e77c6dc4c81886519ca702f6bc9" member_type: VOTER }
I20261120 03:03:36.942318 2734 sys_catalog.cc:565] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:36.945531 2738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48f97e77c6dc4c81886519ca702f6bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f97e77c6dc4c81886519ca702f6bc9" member_type: VOTER } }
I20261120 03:03:36.946187 2738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:36.946025 2739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f97e77c6dc4c81886519ca702f6bc9. Latest consensus state: current_term: 1 leader_uuid: "48f97e77c6dc4c81886519ca702f6bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f97e77c6dc4c81886519ca702f6bc9" member_type: VOTER } }
I20261120 03:03:36.946525 2739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:36.953044 2744 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:36.957643 2744 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:36.961117 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:36.965432 2744 catalog_manager.cc:1398] Generated new cluster ID: b00adcad396f461a8dcbe4c4c06bf074
I20261120 03:03:36.965664 2744 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:36.989125 2744 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:36.990268 2744 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:37.002933 2744 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9: Generated new TSK 0
I20261120 03:03:37.003472 2744 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:37.026790 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:37.032438 2755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:37.033708 2756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:37.035487 2758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:37.035842 385 server_base.cc:1061] running on GCE node
I20261120 03:03:37.036856 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:37.037060 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:37.037237 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143817037220 us; error 0 us; skew 500 ppm
I20261120 03:03:37.037763 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:37.039880 385 webserver.cc:577] Webserver started at http://127.0.96.65:34409/ using document root <none> and password file <none>
I20261120 03:03:37.040369 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:37.040560 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:37.040823 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:37.041916 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "9f2bc43c379a4e998af2b0eee1b31ba6"
format_stamp: "Formatted at 2026-11-20 03:03:37 on dist-test-slave-54pf"
I20261120 03:03:37.046132 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20261120 03:03:37.049027 2763 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:37.049671 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261120 03:03:37.050021 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "9f2bc43c379a4e998af2b0eee1b31ba6"
format_stamp: "Formatted at 2026-11-20 03:03:37 on dist-test-slave-54pf"
I20261120 03:03:37.050290 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:37.066728 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:37.067672 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:37.069028 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:37.070943 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:37.071110 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:37.071269 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:37.071444 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:37.107349 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:39839
I20261120 03:03:37.107445 2825 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:39839 every 8 connection(s)
I20261120 03:03:37.112056 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:37.118384 2830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:37.118839 2831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:37.122540 2833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:37.123454 2826 heartbeater.cc:345] Connected to a master server at 127.0.96.126:35193
I20261120 03:03:37.123790 385 server_base.cc:1061] running on GCE node
I20261120 03:03:37.123868 2826 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:37.124822 2826 heartbeater.cc:508] Master 127.0.96.126:35193 requested a full tablet report, sending...
I20261120 03:03:37.124840 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:37.125238 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:37.125404 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143817125373 us; error 0 us; skew 500 ppm
I20261120 03:03:37.126019 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:37.126928 2699 ts_manager.cc:210] Registered new tserver with Master: 9f2bc43c379a4e998af2b0eee1b31ba6 (127.0.96.65:39839)
I20261120 03:03:37.128551 385 webserver.cc:577] Webserver started at http://127.0.96.66:35119/ using document root <none> and password file <none>
I20261120 03:03:37.128690 2699 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52824
I20261120 03:03:37.129105 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:37.129338 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:37.129614 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:37.130702 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "b6200401445c426b8c6ffe3e81de3b57"
format_stamp: "Formatted at 2026-11-20 03:03:37 on dist-test-slave-54pf"
I20261120 03:03:37.134660 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20261120 03:03:37.137588 2838 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:37.138334 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 03:03:37.138727 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "b6200401445c426b8c6ffe3e81de3b57"
format_stamp: "Formatted at 2026-11-20 03:03:37 on dist-test-slave-54pf"
I20261120 03:03:37.139051 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:37.152740 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:37.153805 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:37.155117 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:37.157248 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:37.157444 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:37.157697 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:37.157872 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:37.194317 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:38703
I20261120 03:03:37.194408 2900 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:38703 every 8 connection(s)
I20261120 03:03:37.198908 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:37.205538 2904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:37.206719 2905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:37.209313 385 server_base.cc:1061] running on GCE node
W20261120 03:03:37.209256 2907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:37.210557 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:37.210769 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:37.210985 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143817210966 us; error 0 us; skew 500 ppm
I20261120 03:03:37.211143 2901 heartbeater.cc:345] Connected to a master server at 127.0.96.126:35193
I20261120 03:03:37.211575 2901 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:37.211764 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:37.212502 2901 heartbeater.cc:508] Master 127.0.96.126:35193 requested a full tablet report, sending...
I20261120 03:03:37.214119 385 webserver.cc:577] Webserver started at http://127.0.96.67:43873/ using document root <none> and password file <none>
I20261120 03:03:37.214645 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:37.214692 2699 ts_manager.cc:210] Registered new tserver with Master: b6200401445c426b8c6ffe3e81de3b57 (127.0.96.66:38703)
I20261120 03:03:37.214955 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:37.215296 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:37.216234 2699 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52834
I20261120 03:03:37.216483 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root/instance:
uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321"
format_stamp: "Formatted at 2026-11-20 03:03:37 on dist-test-slave-54pf"
I20261120 03:03:37.220948 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20261120 03:03:37.223783 2912 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:37.224598 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261120 03:03:37.224889 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root
uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321"
format_stamp: "Formatted at 2026-11-20 03:03:37 on dist-test-slave-54pf"
I20261120 03:03:37.225142 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:37.243182 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:37.244294 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:37.245549 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:37.247562 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:37.247772 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:37.248001 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:37.248190 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20261120 03:03:37.285837 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.67:37007
I20261120 03:03:37.285914 2974 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.67:37007 every 8 connection(s)
I20261120 03:03:37.298624 2975 heartbeater.cc:345] Connected to a master server at 127.0.96.126:35193
I20261120 03:03:37.298906 2975 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:37.299487 2975 heartbeater.cc:508] Master 127.0.96.126:35193 requested a full tablet report, sending...
I20261120 03:03:37.301162 2699 ts_manager.cc:210] Registered new tserver with Master: 4e1e8c6f53ec4ad1bfd398c22a8a7321 (127.0.96.67:37007)
I20261120 03:03:37.302192 385 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01336271s
I20261120 03:03:37.302245 2699 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52848
I20261120 03:03:37.331578 2699 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20261120 03:03:37.333089 2699 catalog_manager.cc:7244] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261120 03:03:37.379233 2866 tablet_service.cc:1495] Processing CreateTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 (DEFAULT_TABLE table=test-table [id=e97d03d945a34c338ddf9162958877e5]), partition=
I20261120 03:03:37.380316 2791 tablet_service.cc:1495] Processing CreateTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 (DEFAULT_TABLE table=test-table [id=e97d03d945a34c338ddf9162958877e5]), partition=
I20261120 03:03:37.380467 2866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62355b6698c54ab2b0ccd94c5749b743. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:37.381314 2791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62355b6698c54ab2b0ccd94c5749b743. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:37.381855 2940 tablet_service.cc:1495] Processing CreateTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 (DEFAULT_TABLE table=test-table [id=e97d03d945a34c338ddf9162958877e5]), partition=
I20261120 03:03:37.382730 2940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62355b6698c54ab2b0ccd94c5749b743. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:37.392630 2991 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Bootstrap starting.
I20261120 03:03:37.397866 2991 tablet_bootstrap.cc:654] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:37.401775 2991 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: No bootstrap required, opened a new log
I20261120 03:03:37.402084 2992 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Bootstrap starting.
I20261120 03:03:37.402215 2991 ts_tablet_manager.cc:1449] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20261120 03:03:37.402385 2993 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Bootstrap starting.
I20261120 03:03:37.404660 2991 raft_consensus.cc:366] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.405289 2991 raft_consensus.cc:392] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:37.405632 2991 raft_consensus.cc:787] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6200401445c426b8c6ffe3e81de3b57, State: Initialized, Role: FOLLOWER
I20261120 03:03:37.406520 2991 consensus_queue.cc:261] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.409225 2992 tablet_bootstrap.cc:654] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:37.409252 2993 tablet_bootstrap.cc:654] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:37.409535 2901 heartbeater.cc:500] Master 127.0.96.126:35193 was elected leader, sending a full tablet report...
I20261120 03:03:37.410061 2991 ts_tablet_manager.cc:1480] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
W20261120 03:03:37.410386 2991 ts_tablet_manager.cc:1503] Injecting 50ms of latency into OpenTablet
I20261120 03:03:37.414786 2992 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: No bootstrap required, opened a new log
I20261120 03:03:37.415238 2992 ts_tablet_manager.cc:1449] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.009s
I20261120 03:03:37.415645 2993 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: No bootstrap required, opened a new log
I20261120 03:03:37.415990 2993 ts_tablet_manager.cc:1449] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20261120 03:03:37.417281 2992 raft_consensus.cc:366] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.417899 2992 raft_consensus.cc:392] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:37.418247 2992 raft_consensus.cc:787] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2bc43c379a4e998af2b0eee1b31ba6, State: Initialized, Role: FOLLOWER
I20261120 03:03:37.418094 2993 raft_consensus.cc:366] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.418895 2993 raft_consensus.cc:392] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:37.419181 2993 raft_consensus.cc:787] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1e8c6f53ec4ad1bfd398c22a8a7321, State: Initialized, Role: FOLLOWER
I20261120 03:03:37.419127 2992 consensus_queue.cc:261] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.420181 2993 consensus_queue.cc:261] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.421643 2826 heartbeater.cc:500] Master 127.0.96.126:35193 was elected leader, sending a full tablet report...
I20261120 03:03:37.422281 2975 heartbeater.cc:500] Master 127.0.96.126:35193 was elected leader, sending a full tablet report...
I20261120 03:03:37.422595 2993 ts_tablet_manager.cc:1480] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20261120 03:03:37.422860 2993 ts_tablet_manager.cc:1503] Injecting 50ms of latency into OpenTablet
I20261120 03:03:37.423707 2992 ts_tablet_manager.cc:1480] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20261120 03:03:37.423967 2992 ts_tablet_manager.cc:1503] Injecting 50ms of latency into OpenTablet
W20261120 03:03:37.447980 2902 tablet.cc:2554] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261120 03:03:37.573923 2995 raft_consensus.cc:534] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:37.574303 2995 raft_consensus.cc:556] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.576249 2995 leader_election.cc:290] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e1e8c6f53ec4ad1bfd398c22a8a7321 (127.0.96.67:37007), 9f2bc43c379a4e998af2b0eee1b31ba6 (127.0.96.65:39839)
I20261120 03:03:37.583871 2950 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "62355b6698c54ab2b0ccd94c5749b743" candidate_uuid: "b6200401445c426b8c6ffe3e81de3b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" is_pre_election: true
I20261120 03:03:37.584713 2950 raft_consensus.cc:2062] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate b6200401445c426b8c6ffe3e81de3b57 in term 0
I20261120 03:03:37.584738 2999 raft_consensus.cc:534] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:37.585492 2999 raft_consensus.cc:556] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.585894 2801 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "62355b6698c54ab2b0ccd94c5749b743" candidate_uuid: "b6200401445c426b8c6ffe3e81de3b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" is_pre_election: true
I20261120 03:03:37.586064 2840 leader_election.cc:304] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e1e8c6f53ec4ad1bfd398c22a8a7321, b6200401445c426b8c6ffe3e81de3b57; no voters:
I20261120 03:03:37.586648 2801 raft_consensus.cc:2062] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate b6200401445c426b8c6ffe3e81de3b57 in term 0
I20261120 03:03:37.586925 2995 raft_consensus.cc:3024] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:37.587292 2995 raft_consensus.cc:534] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:37.587633 2995 raft_consensus.cc:3296] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:37.588024 2999 leader_election.cc:290] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f2bc43c379a4e998af2b0eee1b31ba6 (127.0.96.65:39839), b6200401445c426b8c6ffe3e81de3b57 (127.0.96.66:38703)
I20261120 03:03:37.590991 2995 raft_consensus.cc:556] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.597860 2801 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "62355b6698c54ab2b0ccd94c5749b743" candidate_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" is_pre_election: true
I20261120 03:03:37.598455 2801 raft_consensus.cc:2062] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 4e1e8c6f53ec4ad1bfd398c22a8a7321 in term 0
I20261120 03:03:37.598994 2995 leader_election.cc:290] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [CANDIDATE]: Term 1 election: Requested vote from peers 4e1e8c6f53ec4ad1bfd398c22a8a7321 (127.0.96.67:37007), 9f2bc43c379a4e998af2b0eee1b31ba6 (127.0.96.65:39839)
I20261120 03:03:37.599531 2801 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "62355b6698c54ab2b0ccd94c5749b743" candidate_uuid: "b6200401445c426b8c6ffe3e81de3b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6"
I20261120 03:03:37.600111 2801 raft_consensus.cc:3296] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:37.599879 2876 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "62355b6698c54ab2b0ccd94c5749b743" candidate_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6200401445c426b8c6ffe3e81de3b57" is_pre_election: true
I20261120 03:03:37.600481 2950 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "62355b6698c54ab2b0ccd94c5749b743" candidate_uuid: "b6200401445c426b8c6ffe3e81de3b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321"
I20261120 03:03:37.601058 2950 raft_consensus.cc:3296] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:37.601157 2801 raft_consensus.cc:2062] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate b6200401445c426b8c6ffe3e81de3b57 in term 1
I20261120 03:03:37.601967 2950 raft_consensus.cc:2062] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate b6200401445c426b8c6ffe3e81de3b57 in term 1
I20261120 03:03:37.599869 2913 leader_election.cc:304] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e1e8c6f53ec4ad1bfd398c22a8a7321, 9f2bc43c379a4e998af2b0eee1b31ba6; no voters:
I20261120 03:03:37.603581 3003 raft_consensus.cc:2984] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20261120 03:03:37.600992 2876 raft_consensus.cc:2628] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e1e8c6f53ec4ad1bfd398c22a8a7321 in current term 1: Already voted for candidate b6200401445c426b8c6ffe3e81de3b57 in this term.
I20261120 03:03:37.608165 2839 leader_election.cc:304] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f2bc43c379a4e998af2b0eee1b31ba6, b6200401445c426b8c6ffe3e81de3b57; no voters:
I20261120 03:03:37.609030 2995 raft_consensus.cc:3024] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:37.609443 2995 raft_consensus.cc:744] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 1 LEADER]: Becoming Leader. State: Replica: b6200401445c426b8c6ffe3e81de3b57, State: Running, Role: LEADER
I20261120 03:03:37.610018 2995 consensus_queue.cc:238] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.617424 2995 consensus_queue.cc:1049] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:37.617172 2699 catalog_manager.cc:5856] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6200401445c426b8c6ffe3e81de3b57 (127.0.96.66). New cstate: current_term: 1 leader_uuid: "b6200401445c426b8c6ffe3e81de3b57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:37.618245 3002 consensus_queue.cc:1049] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:37.702065 2940 tablet_service.cc:1542] Processing DeleteTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:36696
I20261120 03:03:37.703416 3012 tablet_replica.cc:333] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: stopping tablet replica
I20261120 03:03:37.703969 3012 raft_consensus.cc:2440] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:37.704459 3012 raft_consensus.cc:2469] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:37.706770 3012 ts_tablet_manager.cc:1962] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20261120 03:03:37.716476 3012 ts_tablet_manager.cc:1975] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20261120 03:03:37.716735 3012 log.cc:1199] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Deleting WAL directory at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root/wals/62355b6698c54ab2b0ccd94c5749b743
I20261120 03:03:37.720441 3013 ts_tablet_manager.cc:979] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Initiating tablet copy from peer b6200401445c426b8c6ffe3e81de3b57 (127.0.96.66:38703)
I20261120 03:03:37.722015 3013 tablet_copy_client.cc:323] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet copy: Beginning tablet copy session from remote peer at address 127.0.96.66:38703
I20261120 03:03:37.723676 2886 tablet_copy_service.cc:140] P b6200401445c426b8c6ffe3e81de3b57: Received BeginTabletCopySession request for tablet 62355b6698c54ab2b0ccd94c5749b743 from peer 4e1e8c6f53ec4ad1bfd398c22a8a7321 ({username='slave'} at 127.0.0.1:46550)
I20261120 03:03:37.724087 2886 tablet_copy_service.cc:161] P b6200401445c426b8c6ffe3e81de3b57: Beginning new tablet copy session on tablet 62355b6698c54ab2b0ccd94c5749b743 from peer 4e1e8c6f53ec4ad1bfd398c22a8a7321 at {username='slave'} at 127.0.0.1:46550: session id = 4e1e8c6f53ec4ad1bfd398c22a8a7321-62355b6698c54ab2b0ccd94c5749b743
I20261120 03:03:37.728626 2886 tablet_copy_source_session.cc:215] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Tablet Copy: opened 0 blocks and 1 log segments
I20261120 03:03:37.730686 3013 ts_tablet_manager.cc:1962] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Deleting tablet data with delete state TABLET_DATA_COPYING
I20261120 03:03:37.739602 3013 ts_tablet_manager.cc:1975] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20261120 03:03:37.740206 3013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62355b6698c54ab2b0ccd94c5749b743. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:37.745079 3013 tablet_copy_client.cc:806] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet copy: Starting download of 0 data blocks...
I20261120 03:03:37.745543 3013 tablet_copy_client.cc:670] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet copy: Starting download of 1 WAL segments...
I20261120 03:03:37.746847 2699 catalog_manager.cc:2551] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52860:
table { table_id: "e97d03d945a34c338ddf9162958877e5" table_name: "test-table" }
I20261120 03:03:37.747483 2699 catalog_manager.cc:2799] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52860:
table { table_id: "e97d03d945a34c338ddf9162958877e5" table_name: "test-table" }
I20261120 03:03:37.748966 3013 tablet_copy_client.cc:538] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261120 03:03:37.754326 3013 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Bootstrap starting.
I20261120 03:03:37.757014 2699 catalog_manager.cc:6151] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9: Sending DeleteTablet for 3 replicas of tablet 62355b6698c54ab2b0ccd94c5749b743
I20261120 03:03:37.759042 2940 tablet_service.cc:1542] Processing DeleteTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-20 03:03:37 UTC) from {username='slave'} at 127.0.0.1:36680
I20261120 03:03:37.759325 2866 tablet_service.cc:1542] Processing DeleteTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-20 03:03:37 UTC) from {username='slave'} at 127.0.0.1:46534
I20261120 03:03:37.759738 2791 tablet_service.cc:1542] Processing DeleteTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-20 03:03:37 UTC) from {username='slave'} at 127.0.0.1:49972
W20261120 03:03:37.761077 2684 catalog_manager.cc:5061] TS 4e1e8c6f53ec4ad1bfd398c22a8a7321 (127.0.96.67:37007): delete failed for tablet 62355b6698c54ab2b0ccd94c5749b743 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62355b6698c54ab2b0ccd94c5749b743 already in progress: copying tablet
I20261120 03:03:37.763185 3015 tablet_replica.cc:333] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: stopping tablet replica
I20261120 03:03:37.763933 3016 tablet_replica.cc:333] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: stopping tablet replica
I20261120 03:03:37.763952 3015 raft_consensus.cc:2440] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:37.764940 3016 raft_consensus.cc:2440] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:37.765059 3015 raft_consensus.cc:2469] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:37.766001 3016 raft_consensus.cc:2469] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:37.768770 3016 ts_tablet_manager.cc:1962] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261120 03:03:37.769438 3015 ts_tablet_manager.cc:1962] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261120 03:03:37.776947 3013 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:37.777750 3013 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Bootstrap complete.
I20261120 03:03:37.778242 3016 ts_tablet_manager.cc:1975] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261120 03:03:37.778398 3013 ts_tablet_manager.cc:1449] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.008s
I20261120 03:03:37.778694 3016 log.cc:1199] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Deleting WAL directory at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-0-root/wals/62355b6698c54ab2b0ccd94c5749b743
I20261120 03:03:37.779461 3016 ts_tablet_manager.cc:1996] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Deleting consensus metadata
I20261120 03:03:37.781761 2683 catalog_manager.cc:5081] TS 9f2bc43c379a4e998af2b0eee1b31ba6 (127.0.96.65:39839): tablet 62355b6698c54ab2b0ccd94c5749b743 (table test-table [id=e97d03d945a34c338ddf9162958877e5]) successfully deleted
I20261120 03:03:37.781464 3013 raft_consensus.cc:366] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.782209 3013 raft_consensus.cc:787] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1e8c6f53ec4ad1bfd398c22a8a7321, State: Initialized, Role: FOLLOWER
I20261120 03:03:37.782871 3013 consensus_queue.cc:261] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.783362 3015 ts_tablet_manager.cc:1975] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261120 03:03:37.783725 3015 log.cc:1199] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Deleting WAL directory at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-1-root/wals/62355b6698c54ab2b0ccd94c5749b743
I20261120 03:03:37.784483 3015 ts_tablet_manager.cc:1996] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Deleting consensus metadata
I20261120 03:03:37.784538 3013 ts_tablet_manager.cc:1480] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20261120 03:03:37.784991 3013 ts_tablet_manager.cc:1503] Injecting 50ms of latency into OpenTablet
I20261120 03:03:37.786028 2685 catalog_manager.cc:5081] TS b6200401445c426b8c6ffe3e81de3b57 (127.0.96.66:38703): tablet 62355b6698c54ab2b0ccd94c5749b743 (table test-table [id=e97d03d945a34c338ddf9162958877e5]) successfully deleted
I20261120 03:03:37.788815 2940 tablet_service.cc:1542] Processing DeleteTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-20 03:03:37 UTC) from {username='slave'} at 127.0.0.1:36680
W20261120 03:03:37.790019 2684 catalog_manager.cc:5061] TS 4e1e8c6f53ec4ad1bfd398c22a8a7321 (127.0.96.67:37007): delete failed for tablet 62355b6698c54ab2b0ccd94c5749b743 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62355b6698c54ab2b0ccd94c5749b743 already in progress: copying tablet
I20261120 03:03:37.836783 2886 tablet_copy_service.cc:342] P b6200401445c426b8c6ffe3e81de3b57: Request end of tablet copy session 4e1e8c6f53ec4ad1bfd398c22a8a7321-62355b6698c54ab2b0ccd94c5749b743 received from {username='slave'} at 127.0.0.1:46550
I20261120 03:03:37.837286 2886 tablet_copy_service.cc:434] P b6200401445c426b8c6ffe3e81de3b57: ending tablet copy session 4e1e8c6f53ec4ad1bfd398c22a8a7321-62355b6698c54ab2b0ccd94c5749b743 on tablet 62355b6698c54ab2b0ccd94c5749b743 with peer 4e1e8c6f53ec4ad1bfd398c22a8a7321
I20261120 03:03:37.840396 2940 tablet_service.cc:1542] Processing DeleteTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-20 03:03:37 UTC) from {username='slave'} at 127.0.0.1:36680
I20261120 03:03:37.841112 3012 tablet_replica.cc:333] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: stopping tablet replica
I20261120 03:03:37.841753 3012 raft_consensus.cc:2440] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:37.842216 3012 raft_consensus.cc:2469] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:37.844332 3012 ts_tablet_manager.cc:1962] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261120 03:03:37.854117 3012 ts_tablet_manager.cc:1975] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261120 03:03:37.854394 3012 log.cc:1199] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Deleting WAL directory at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root/wals/62355b6698c54ab2b0ccd94c5749b743
I20261120 03:03:37.854986 3012 ts_tablet_manager.cc:1996] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Deleting consensus metadata
I20261120 03:03:37.856973 2684 catalog_manager.cc:5081] TS 4e1e8c6f53ec4ad1bfd398c22a8a7321 (127.0.96.67:37007): tablet 62355b6698c54ab2b0ccd94c5749b743 (table test-table [id=e97d03d945a34c338ddf9162958877e5]) successfully deleted
I20261120 03:03:37.893677 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:37.911146 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:37.927110 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:37.935081 385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:37.949620 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:37.956969 385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:37.964517 385 tablet_server.cc:179] TabletServer@127.0.96.67:0 shutting down...
I20261120 03:03:37.979135 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:37.996016 385 tablet_server.cc:196] TabletServer@127.0.96.67:0 shutdown complete.
I20261120 03:03:38.004001 385 master.cc:585] Master@127.0.96.126:35193 shutting down...
I20261120 03:03:38.016253 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:38.016665 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:38.016921 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9: stopping tablet replica
I20261120 03:03:38.024362 385 master.cc:607] Master@127.0.96.126:35193 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1224 ms)
[----------] 4 tests from TsTabletManagerITest (43818 ms total)
[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20261120 03:03:38.046993 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:33613
I20261120 03:03:38.047815 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:38.052239 3022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.053422 3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.054831 3025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:38.055509 385 server_base.cc:1061] running on GCE node
I20261120 03:03:38.056279 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:38.056442 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:38.056550 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143818056539 us; error 0 us; skew 500 ppm
I20261120 03:03:38.056965 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:38.058912 385 webserver.cc:577] Webserver started at http://127.0.96.126:33819/ using document root <none> and password file <none>
I20261120 03:03:38.059326 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:38.059474 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:38.059664 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:38.060644 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.064630 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20261120 03:03:38.067379 3030 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:38.068064 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261120 03:03:38.068390 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.068702 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:38.095054 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:38.095896 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:38.127470 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:33613
I20261120 03:03:38.127617 3081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:33613 every 8 connection(s)
I20261120 03:03:38.130923 3082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:38.140024 3082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d: Bootstrap starting.
I20261120 03:03:38.143985 3082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:38.147599 3082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d: No bootstrap required, opened a new log
I20261120 03:03:38.149374 3082 raft_consensus.cc:366] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" member_type: VOTER }
I20261120 03:03:38.149698 3082 raft_consensus.cc:392] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:38.149976 3082 raft_consensus.cc:787] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a9a0480eb54b3f8507798a5b4f1d7d, State: Initialized, Role: FOLLOWER
I20261120 03:03:38.150527 3082 consensus_queue.cc:261] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" member_type: VOTER }
I20261120 03:03:38.150995 3082 raft_consensus.cc:407] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:38.151206 3082 raft_consensus.cc:534] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:38.151487 3082 raft_consensus.cc:3296] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:38.152544 3082 raft_consensus.cc:556] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" member_type: VOTER }
I20261120 03:03:38.156459 3082 leader_election.cc:304] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0a9a0480eb54b3f8507798a5b4f1d7d; no voters:
I20261120 03:03:38.156924 3082 leader_election.cc:290] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:38.157058 3085 raft_consensus.cc:3024] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:38.158406 3085 raft_consensus.cc:744] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 1 LEADER]: Becoming Leader. State: Replica: d0a9a0480eb54b3f8507798a5b4f1d7d, State: Running, Role: LEADER
I20261120 03:03:38.158990 3085 consensus_queue.cc:238] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" member_type: VOTER }
I20261120 03:03:38.159837 3082 sys_catalog.cc:565] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:38.161417 3086 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" member_type: VOTER } }
I20261120 03:03:38.161444 3087 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0a9a0480eb54b3f8507798a5b4f1d7d. Latest consensus state: current_term: 1 leader_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" member_type: VOTER } }
I20261120 03:03:38.162071 3086 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:38.162211 3087 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:38.164747 3089 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:38.169283 3089 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:38.174937 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:38.177464 3089 catalog_manager.cc:1398] Generated new cluster ID: 7d429e6644994dc8b3a50487f73512e6
I20261120 03:03:38.177699 3089 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:38.194279 3089 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:38.195394 3089 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:38.207247 3089 catalog_manager.cc:6237] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d: Generated new TSK 0
I20261120 03:03:38.207782 3089 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:38.241671 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:38.246934 3103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.247908 3104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.249339 3106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:38.249595 385 server_base.cc:1061] running on GCE node
I20261120 03:03:38.250379 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:38.250617 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:38.250833 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143818250813 us; error 0 us; skew 500 ppm
I20261120 03:03:38.251739 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:38.253851 385 webserver.cc:577] Webserver started at http://127.0.96.65:44629/ using document root <none> and password file <none>
I20261120 03:03:38.254345 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:38.254554 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:38.254818 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:38.255822 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "2b4b6609ec75466faa98fdf15bc21513"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.259929 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20261120 03:03:38.262929 3111 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:38.263665 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 03:03:38.263950 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "2b4b6609ec75466faa98fdf15bc21513"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.264287 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:38.279279 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:38.280294 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:38.284619 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:38.284965 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20261120 03:03:38.285368 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:38.285679 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:38.333323 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:37323
I20261120 03:03:38.333424 3181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:37323 every 8 connection(s)
I20261120 03:03:38.337821 3182 heartbeater.cc:345] Connected to a master server at 127.0.96.126:33613
I20261120 03:03:38.338173 3182 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:38.338856 3182 heartbeater.cc:508] Master 127.0.96.126:33613 requested a full tablet report, sending...
I20261120 03:03:38.340669 3047 ts_manager.cc:210] Registered new tserver with Master: 2b4b6609ec75466faa98fdf15bc21513 (127.0.96.65:37323)
I20261120 03:03:38.341233 385 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004807337s
I20261120 03:03:38.342628 3047 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34144
I20261120 03:03:38.369163 3144 tablet_service.cc:1495] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261120 03:03:38.370262 3144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:38.380915 3192 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Bootstrap starting.
I20261120 03:03:38.385248 3192 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:38.388710 3192 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: No bootstrap required, opened a new log
I20261120 03:03:38.389050 3192 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20261120 03:03:38.390914 3192 raft_consensus.cc:366] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.391351 3192 raft_consensus.cc:392] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:38.391592 3192 raft_consensus.cc:787] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4b6609ec75466faa98fdf15bc21513, State: Initialized, Role: FOLLOWER
I20261120 03:03:38.392146 3192 consensus_queue.cc:261] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.392652 3192 raft_consensus.cc:407] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:38.392879 3192 raft_consensus.cc:534] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:38.393142 3192 raft_consensus.cc:3296] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:38.394240 3192 raft_consensus.cc:556] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.398432 3192 leader_election.cc:304] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b4b6609ec75466faa98fdf15bc21513; no voters:
I20261120 03:03:38.398914 3192 leader_election.cc:290] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:38.399055 3194 raft_consensus.cc:3024] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:38.400314 3194 raft_consensus.cc:744] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 LEADER]: Becoming Leader. State: Replica: 2b4b6609ec75466faa98fdf15bc21513, State: Running, Role: LEADER
I20261120 03:03:38.400795 3182 heartbeater.cc:500] Master 127.0.96.126:33613 was elected leader, sending a full tablet report...
I20261120 03:03:38.401002 3194 consensus_queue.cc:238] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.401332 3192 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20261120 03:03:38.404860 3196 tablet_replica.cc:442] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: TxnStatusTablet state changed. Reason: New leader 2b4b6609ec75466faa98fdf15bc21513. Latest consensus state: current_term: 1 leader_uuid: "2b4b6609ec75466faa98fdf15bc21513" committed_config { opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } } }
I20261120 03:03:38.405422 3196 tablet_replica.cc:445] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:38.406159 3195 tablet_replica.cc:442] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b4b6609ec75466faa98fdf15bc21513" committed_config { opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } } }
I20261120 03:03:38.406800 3195 tablet_replica.cc:445] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: This TxnStatusTablet replica's current role is: LEADER
W20261120 03:03:38.408792 3047 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:38.411069 3199 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261120 03:03:38.411535 3199 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20261120 03:03:38.413298 3047 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:38.444834 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:38.467916 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:38.468705 385 tablet_replica.cc:333] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: stopping tablet replica
I20261120 03:03:38.469285 385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:38.469785 385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:38.491005 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:38.501700 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:38.507189 3202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.507602 3203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:38.510063 385 server_base.cc:1061] running on GCE node
W20261120 03:03:38.510900 3206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:38.511818 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:38.512076 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:38.512297 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143818512279 us; error 0 us; skew 500 ppm
I20261120 03:03:38.513002 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:38.515527 385 webserver.cc:577] Webserver started at http://127.0.96.65:44629/ using document root <none> and password file <none>
I20261120 03:03:38.516067 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:38.516287 385 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261120 03:03:38.520737 385 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20261120 03:03:38.523923 3210 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:38.525939 385 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20261120 03:03:38.526365 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "2b4b6609ec75466faa98fdf15bc21513"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.526681 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:38.538625 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:38.539753 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:38.547371 3219 ts_tablet_manager.cc:594] Loading tablet metadata (0/1 complete)
I20261120 03:03:38.554382 385 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261120 03:03:38.554675 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.001s
I20261120 03:03:38.555027 385 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261120 03:03:38.561659 385 ts_tablet_manager.cc:662] Registered 1 tablets
I20261120 03:03:38.561928 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.007s user 0.007s sys 0.000s
I20261120 03:03:38.562053 3219 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Bootstrap starting.
I20261120 03:03:38.603374 3219 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:38.604401 3219 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Bootstrap complete.
I20261120 03:03:38.605239 3219 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Time spent bootstrapping tablet: real 0.043s user 0.030s sys 0.012s
I20261120 03:03:38.608026 3219 raft_consensus.cc:366] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.608671 3219 raft_consensus.cc:787] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4b6609ec75466faa98fdf15bc21513, State: Initialized, Role: FOLLOWER
I20261120 03:03:38.609582 3219 consensus_queue.cc:261] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.610260 3219 raft_consensus.cc:407] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:38.610569 3219 raft_consensus.cc:534] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:38.610997 3219 raft_consensus.cc:3296] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:38.623193 3219 raft_consensus.cc:556] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.623845 3219 leader_election.cc:304] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b4b6609ec75466faa98fdf15bc21513; no voters:
I20261120 03:03:38.624598 3219 leader_election.cc:290] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [CANDIDATE]: Term 2 election: Requested vote from peers
I20261120 03:03:38.625162 3251 raft_consensus.cc:3024] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:38.631089 3219 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Time spent starting tablet: real 0.025s user 0.011s sys 0.007s
I20261120 03:03:38.633127 3251 raft_consensus.cc:744] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 2 LEADER]: Becoming Leader. State: Replica: 2b4b6609ec75466faa98fdf15bc21513, State: Running, Role: LEADER
I20261120 03:03:38.633842 3251 consensus_queue.cc:238] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.636710 3264 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b4b6609ec75466faa98fdf15bc21513" committed_config { opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } } }
I20261120 03:03:38.636857 3265 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 2b4b6609ec75466faa98fdf15bc21513. Latest consensus state: current_term: 2 leader_uuid: "2b4b6609ec75466faa98fdf15bc21513" committed_config { opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } } }
I20261120 03:03:38.637264 3265 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:38.637217 3264 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:38.639274 3269 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261120 03:03:38.639799 3269 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20261120 03:03:38.665566 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:37323
I20261120 03:03:38.665700 3287 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:37323 every 8 connection(s)
I20261120 03:03:38.671871 3288 heartbeater.cc:345] Connected to a master server at 127.0.96.126:33613
I20261120 03:03:38.672273 3288 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:38.672922 3288 heartbeater.cc:508] Master 127.0.96.126:33613 requested a full tablet report, sending...
I20261120 03:03:38.673813 385 tablet_server.cc:179] TabletServer@127.0.96.65:37323 shutting down...
I20261120 03:03:38.676519 3047 ts_manager.cc:210] Re-registered known tserver with Master: 2b4b6609ec75466faa98fdf15bc21513 (127.0.96.65:37323)
W20261120 03:03:38.677172 3047 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:38.679832 3047 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34174
I20261120 03:03:38.682637 3288 heartbeater.cc:500] Master 127.0.96.126:33613 was elected leader, sending a full tablet report...
W20261120 03:03:38.685861 3047 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:38.697319 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:38.698107 385 tablet_replica.cc:333] stopping tablet replica
I20261120 03:03:38.698545 385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 2 LEADER]: Raft consensus shutting down.
I20261120 03:03:38.698938 385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:38.709548 385 tablet_server.cc:196] TabletServer@127.0.96.65:37323 shutdown complete.
I20261120 03:03:38.719607 385 master.cc:585] Master@127.0.96.126:33613 shutting down...
I20261120 03:03:38.736292 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:38.736702 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:38.737025 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d: stopping tablet replica
I20261120 03:03:38.754426 385 master.cc:607] Master@127.0.96.126:33613 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (728 ms)
[ RUN ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20261120 03:03:38.777001 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:44779
I20261120 03:03:38.777968 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:38.782974 3294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.784624 3295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.785967 3297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:38.786686 385 server_base.cc:1061] running on GCE node
I20261120 03:03:38.787498 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:38.787734 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:38.787912 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143818787898 us; error 0 us; skew 500 ppm
I20261120 03:03:38.788625 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:38.791148 385 webserver.cc:577] Webserver started at http://127.0.96.126:34183/ using document root <none> and password file <none>
I20261120 03:03:38.791749 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:38.791965 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:38.792290 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:38.793642 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "90084b6d6a1149049fe264596414f97f"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.799250 385 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20261120 03:03:38.803155 3302 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:38.804008 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 03:03:38.804379 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "90084b6d6a1149049fe264596414f97f"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.804715 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:38.839732 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:38.841032 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:38.883744 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:44779
I20261120 03:03:38.883816 3353 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:44779 every 8 connection(s)
I20261120 03:03:38.888427 3354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:38.897773 3354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f: Bootstrap starting.
I20261120 03:03:38.901922 3354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:38.905615 3354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f: No bootstrap required, opened a new log
I20261120 03:03:38.907755 3354 raft_consensus.cc:366] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90084b6d6a1149049fe264596414f97f" member_type: VOTER }
I20261120 03:03:38.908149 3354 raft_consensus.cc:392] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:38.908399 3354 raft_consensus.cc:787] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90084b6d6a1149049fe264596414f97f, State: Initialized, Role: FOLLOWER
I20261120 03:03:38.908974 3354 consensus_queue.cc:261] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90084b6d6a1149049fe264596414f97f" member_type: VOTER }
I20261120 03:03:38.909403 3354 raft_consensus.cc:407] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:38.909636 3354 raft_consensus.cc:534] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:38.909888 3354 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:38.911149 3354 raft_consensus.cc:556] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90084b6d6a1149049fe264596414f97f" member_type: VOTER }
I20261120 03:03:38.915311 3354 leader_election.cc:304] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90084b6d6a1149049fe264596414f97f; no voters:
I20261120 03:03:38.915831 3354 leader_election.cc:290] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:38.915951 3357 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:38.917809 3357 raft_consensus.cc:744] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 1 LEADER]: Becoming Leader. State: Replica: 90084b6d6a1149049fe264596414f97f, State: Running, Role: LEADER
I20261120 03:03:38.918542 3357 consensus_queue.cc:238] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90084b6d6a1149049fe264596414f97f" member_type: VOTER }
I20261120 03:03:38.919188 3354 sys_catalog.cc:565] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:38.921428 3358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90084b6d6a1149049fe264596414f97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90084b6d6a1149049fe264596414f97f" member_type: VOTER } }
I20261120 03:03:38.921506 3359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90084b6d6a1149049fe264596414f97f. Latest consensus state: current_term: 1 leader_uuid: "90084b6d6a1149049fe264596414f97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90084b6d6a1149049fe264596414f97f" member_type: VOTER } }
I20261120 03:03:38.921998 3358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:38.922163 3359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:38.923926 3362 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:38.932618 3362 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:38.947317 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:38.948166 3362 catalog_manager.cc:1398] Generated new cluster ID: 3d21ad82e2d44867b991835b46a85e85
I20261120 03:03:38.948463 3362 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:38.977007 3362 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:38.978188 3362 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:38.993410 3362 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f: Generated new TSK 0
I20261120 03:03:38.993928 3362 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:39.013108 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:39.019004 3375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:39.020277 3376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:39.022747 3378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:39.023042 385 server_base.cc:1061] running on GCE node
I20261120 03:03:39.023939 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:39.024176 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:39.024366 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143819024347 us; error 0 us; skew 500 ppm
I20261120 03:03:39.024910 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:39.027498 385 webserver.cc:577] Webserver started at http://127.0.96.65:33121/ using document root <none> and password file <none>
I20261120 03:03:39.027962 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:39.028180 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:39.028451 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:39.029439 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "29f19b523fc04e318d7afa0b6a545ab0"
format_stamp: "Formatted at 2026-11-20 03:03:39 on dist-test-slave-54pf"
I20261120 03:03:39.034052 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20261120 03:03:39.037467 3383 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:39.038319 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261120 03:03:39.038600 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "29f19b523fc04e318d7afa0b6a545ab0"
format_stamp: "Formatted at 2026-11-20 03:03:39 on dist-test-slave-54pf"
I20261120 03:03:39.038880 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:39.056347 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:39.057201 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:39.069103 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:39.069528 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:39.069921 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:39.070230 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:39.134486 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:43325
I20261120 03:03:39.134596 3453 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:43325 every 8 connection(s)
I20261120 03:03:39.138975 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261120 03:03:39.139176 3454 heartbeater.cc:345] Connected to a master server at 127.0.96.126:44779
I20261120 03:03:39.139545 3454 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:39.140400 3454 heartbeater.cc:508] Master 127.0.96.126:44779 requested a full tablet report, sending...
I20261120 03:03:39.142834 3319 ts_manager.cc:210] Registered new tserver with Master: 29f19b523fc04e318d7afa0b6a545ab0 (127.0.96.65:43325)
I20261120 03:03:39.145284 3319 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51690
W20261120 03:03:39.145722 3456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:39.148631 3457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:39.151209 385 server_base.cc:1061] running on GCE node
W20261120 03:03:39.151683 3459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:39.152599 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:39.152792 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:39.152961 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143819152945 us; error 0 us; skew 500 ppm
I20261120 03:03:39.153401 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:39.155987 385 webserver.cc:577] Webserver started at http://127.0.96.66:43279/ using document root <none> and password file <none>
I20261120 03:03:39.156498 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:39.156669 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:39.156922 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:39.157923 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "2bc2e534f6364cfc80a46267fed30937"
format_stamp: "Formatted at 2026-11-20 03:03:39 on dist-test-slave-54pf"
I20261120 03:03:39.162351 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20261120 03:03:39.165693 3464 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:39.166433 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 03:03:39.166700 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "2bc2e534f6364cfc80a46267fed30937"
format_stamp: "Formatted at 2026-11-20 03:03:39 on dist-test-slave-54pf"
I20261120 03:03:39.166980 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:39.177129 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:39.178004 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:39.183730 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:39.184079 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20261120 03:03:39.184404 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:39.184581 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:39.301927 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:42795
I20261120 03:03:39.302165 3534 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:42795 every 8 connection(s)
I20261120 03:03:39.309206 3535 heartbeater.cc:345] Connected to a master server at 127.0.96.126:44779
I20261120 03:03:39.309597 3535 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:39.310452 3535 heartbeater.cc:508] Master 127.0.96.126:44779 requested a full tablet report, sending...
I20261120 03:03:39.312848 3319 ts_manager.cc:210] Registered new tserver with Master: 2bc2e534f6364cfc80a46267fed30937 (127.0.96.66:42795)
I20261120 03:03:39.313560 385 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.006529321s
I20261120 03:03:39.314637 3319 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51696
I20261120 03:03:39.352620 3419 tablet_service.cc:1495] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261120 03:03:39.354125 3419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:39.366775 3545 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Bootstrap starting.
I20261120 03:03:39.371899 3545 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:39.376322 3545 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: No bootstrap required, opened a new log
I20261120 03:03:39.376689 3545 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20261120 03:03:39.378970 3545 raft_consensus.cc:366] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.379684 3545 raft_consensus.cc:392] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:39.379925 3545 raft_consensus.cc:787] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f19b523fc04e318d7afa0b6a545ab0, State: Initialized, Role: FOLLOWER
I20261120 03:03:39.380532 3545 consensus_queue.cc:261] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.381021 3545 raft_consensus.cc:407] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:39.381251 3545 raft_consensus.cc:534] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:39.381569 3545 raft_consensus.cc:3296] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:39.382936 3545 raft_consensus.cc:556] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.389446 3545 leader_election.cc:304] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f19b523fc04e318d7afa0b6a545ab0; no voters:
I20261120 03:03:39.389961 3545 leader_election.cc:290] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:39.390188 3547 raft_consensus.cc:3024] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:39.392256 3454 heartbeater.cc:500] Master 127.0.96.126:44779 was elected leader, sending a full tablet report...
I20261120 03:03:39.392671 3545 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Time spent starting tablet: real 0.016s user 0.006s sys 0.004s
I20261120 03:03:39.393440 3547 raft_consensus.cc:744] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 29f19b523fc04e318d7afa0b6a545ab0, State: Running, Role: LEADER
I20261120 03:03:39.394208 3547 consensus_queue.cc:238] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.399405 3549 tablet_replica.cc:442] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: TxnStatusTablet state changed. Reason: New leader 29f19b523fc04e318d7afa0b6a545ab0. Latest consensus state: current_term: 1 leader_uuid: "29f19b523fc04e318d7afa0b6a545ab0" committed_config { opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } } }
I20261120 03:03:39.400182 3549 tablet_replica.cc:445] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: This TxnStatusTablet replica's current role is: LEADER
W20261120 03:03:39.401614 3319 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:39.404492 3548 tablet_replica.cc:442] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29f19b523fc04e318d7afa0b6a545ab0" committed_config { opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } } }
I20261120 03:03:39.405079 3548 tablet_replica.cc:445] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: This TxnStatusTablet replica's current role is: LEADER
W20261120 03:03:39.413125 3319 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:39.413573 3552 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261120 03:03:39.414068 3552 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20261120 03:03:39.466661 3556 ts_tablet_manager.cc:979] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: Initiating tablet copy from peer 29f19b523fc04e318d7afa0b6a545ab0 (127.0.96.65:43325)
I20261120 03:03:39.468468 3556 tablet_copy_client.cc:323] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: tablet copy: Beginning tablet copy session from remote peer at address 127.0.96.65:43325
I20261120 03:03:39.478209 3439 tablet_copy_service.cc:140] P 29f19b523fc04e318d7afa0b6a545ab0: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer 2bc2e534f6364cfc80a46267fed30937 ({username='slave'} at 127.0.0.1:33944)
I20261120 03:03:39.478766 3439 tablet_copy_service.cc:161] P 29f19b523fc04e318d7afa0b6a545ab0: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer 2bc2e534f6364cfc80a46267fed30937 at {username='slave'} at 127.0.0.1:33944: session id = 2bc2e534f6364cfc80a46267fed30937-11111111111111111111111111111111
I20261120 03:03:39.486583 3439 tablet_copy_source_session.cc:215] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Tablet Copy: opened 0 blocks and 1 log segments
I20261120 03:03:39.489581 3556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:39.501308 3556 tablet_copy_client.cc:806] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: tablet copy: Starting download of 0 data blocks...
I20261120 03:03:39.501874 3556 tablet_copy_client.cc:670] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: tablet copy: Starting download of 1 WAL segments...
I20261120 03:03:39.505625 3556 tablet_copy_client.cc:538] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261120 03:03:39.511534 3556 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: Bootstrap starting.
I20261120 03:03:39.535190 3556 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:39.535859 3556 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: Bootstrap complete.
I20261120 03:03:39.536470 3556 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.005s
I20261120 03:03:39.538587 3556 raft_consensus.cc:366] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.538949 3556 raft_consensus.cc:787] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 2bc2e534f6364cfc80a46267fed30937, State: Initialized, Role: NON_PARTICIPANT
I20261120 03:03:39.539464 3556 consensus_queue.cc:261] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.541438 3535 heartbeater.cc:500] Master 127.0.96.126:44779 was elected leader, sending a full tablet report...
I20261120 03:03:39.541457 3559 tablet_replica.cc:442] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } } }
I20261120 03:03:39.541993 3556 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20261120 03:03:39.542106 3559 tablet_replica.cc:445] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20261120 03:03:39.543777 3439 tablet_copy_service.cc:342] P 29f19b523fc04e318d7afa0b6a545ab0: Request end of tablet copy session 2bc2e534f6364cfc80a46267fed30937-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:33944
I20261120 03:03:39.544296 3439 tablet_copy_service.cc:434] P 29f19b523fc04e318d7afa0b6a545ab0: ending tablet copy session 2bc2e534f6364cfc80a46267fed30937-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer 2bc2e534f6364cfc80a46267fed30937
W20261120 03:03:39.544634 3319 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
W20261120 03:03:39.548262 3319 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:39.566248 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:39.591985 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:39.592936 385 tablet_replica.cc:333] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: stopping tablet replica
I20261120 03:03:39.593420 385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:39.593847 385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:39.614725 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:39.627701 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:39.633776 3560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:39.635381 3561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:39.637706 3563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:39.637914 385 server_base.cc:1061] running on GCE node
I20261120 03:03:39.638983 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:39.639195 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:39.639429 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143819639413 us; error 0 us; skew 500 ppm
I20261120 03:03:39.639978 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:39.642980 385 webserver.cc:577] Webserver started at http://127.0.96.65:33121/ using document root <none> and password file <none>
I20261120 03:03:39.643568 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:39.643781 385 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261120 03:03:39.648080 385 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20261120 03:03:39.651216 3568 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:39.652027 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 03:03:39.652382 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "29f19b523fc04e318d7afa0b6a545ab0"
format_stamp: "Formatted at 2026-11-20 03:03:39 on dist-test-slave-54pf"
I20261120 03:03:39.652789 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:39.699798 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:39.701328 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:39.709100 3577 ts_tablet_manager.cc:594] Loading tablet metadata (0/1 complete)
I20261120 03:03:39.716747 385 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261120 03:03:39.717092 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20261120 03:03:39.717418 385 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261120 03:03:39.723649 385 ts_tablet_manager.cc:662] Registered 1 tablets
I20261120 03:03:39.723934 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.007s user 0.004s sys 0.002s
I20261120 03:03:39.723986 3577 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Bootstrap starting.
I20261120 03:03:39.774514 3577 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:39.775728 3577 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Bootstrap complete.
I20261120 03:03:39.776535 3577 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Time spent bootstrapping tablet: real 0.053s user 0.038s sys 0.011s
I20261120 03:03:39.780279 3577 raft_consensus.cc:366] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.780925 3577 raft_consensus.cc:787] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f19b523fc04e318d7afa0b6a545ab0, State: Initialized, Role: FOLLOWER
I20261120 03:03:39.781551 3577 consensus_queue.cc:261] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.782045 3577 raft_consensus.cc:407] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:39.782302 3577 raft_consensus.cc:534] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:39.782603 3577 raft_consensus.cc:3296] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:39.783998 3577 raft_consensus.cc:556] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.788662 3577 leader_election.cc:304] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f19b523fc04e318d7afa0b6a545ab0; no voters:
I20261120 03:03:39.789165 3577 leader_election.cc:290] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20261120 03:03:39.789247 3603 raft_consensus.cc:3024] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:39.798045 3577 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Time spent starting tablet: real 0.021s user 0.011s sys 0.003s
I20261120 03:03:39.799409 3603 raft_consensus.cc:744] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 2 LEADER]: Becoming Leader. State: Replica: 29f19b523fc04e318d7afa0b6a545ab0, State: Running, Role: LEADER
I20261120 03:03:39.800287 3603 consensus_queue.cc:238] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.803408 3607 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 29f19b523fc04e318d7afa0b6a545ab0. Latest consensus state: current_term: 2 leader_uuid: "29f19b523fc04e318d7afa0b6a545ab0" committed_config { opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } } }
I20261120 03:03:39.803500 3606 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "29f19b523fc04e318d7afa0b6a545ab0" committed_config { opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } } }
I20261120 03:03:39.803918 3607 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:39.805014 3606 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:39.807024 3612 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261120 03:03:39.807510 3612 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20261120 03:03:39.836958 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:43325
I20261120 03:03:39.837103 3645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:43325 every 8 connection(s)
I20261120 03:03:39.842549 3646 heartbeater.cc:345] Connected to a master server at 127.0.96.126:44779
I20261120 03:03:39.843006 3646 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:39.843951 3646 heartbeater.cc:508] Master 127.0.96.126:44779 requested a full tablet report, sending...
I20261120 03:03:39.845338 385 tablet_server.cc:179] TabletServer@127.0.96.65:43325 shutting down...
I20261120 03:03:39.847255 3319 ts_manager.cc:210] Re-registered known tserver with Master: 29f19b523fc04e318d7afa0b6a545ab0 (127.0.96.65:43325)
W20261120 03:03:39.847904 3319 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:39.849491 3319 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51724
I20261120 03:03:39.853502 3646 heartbeater.cc:500] Master 127.0.96.126:44779 was elected leader, sending a full tablet report...
W20261120 03:03:39.860013 3319 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:39.868870 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:39.869601 385 tablet_replica.cc:333] stopping tablet replica
I20261120 03:03:39.870154 385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 2 LEADER]: Raft consensus shutting down.
I20261120 03:03:39.870601 385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:39.882125 385 tablet_server.cc:196] TabletServer@127.0.96.65:43325 shutdown complete.
I20261120 03:03:39.894556 385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:39.913226 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:39.914215 385 tablet_replica.cc:333] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: stopping tablet replica
I20261120 03:03:39.914847 385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20261120 03:03:39.915282 385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20261120 03:03:39.934993 385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:39.946018 385 master.cc:585] Master@127.0.96.126:44779 shutting down...
I20261120 03:03:39.968099 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:39.968645 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:39.969000 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f: stopping tablet replica
I20261120 03:03:39.987951 385 master.cc:607] Master@127.0.96.126:44779 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1244 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20261120 03:03:40.021370 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:35375
I20261120 03:03:40.022516 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:40.027840 3652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.029366 3653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.030719 3655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:40.031698 385 server_base.cc:1061] running on GCE node
I20261120 03:03:40.032490 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:40.032692 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:40.032866 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143820032850 us; error 0 us; skew 500 ppm
I20261120 03:03:40.033371 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:40.035655 385 webserver.cc:577] Webserver started at http://127.0.96.126:39129/ using document root <none> and password file <none>
I20261120 03:03:40.036157 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:40.036360 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:40.036638 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:40.037850 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "a20fe6cb0199458c8e038b7230fbd4d2"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.042358 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20261120 03:03:40.045689 3660 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:40.046444 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 03:03:40.046707 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "a20fe6cb0199458c8e038b7230fbd4d2"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.047015 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:40.070524 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:40.072292 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:40.120599 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:35375
I20261120 03:03:40.120684 3711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:35375 every 8 connection(s)
I20261120 03:03:40.124970 3712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:40.139037 3712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2: Bootstrap starting.
I20261120 03:03:40.145249 3712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:40.150049 3712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2: No bootstrap required, opened a new log
I20261120 03:03:40.152752 3712 raft_consensus.cc:366] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" member_type: VOTER }
I20261120 03:03:40.153303 3712 raft_consensus.cc:392] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:40.153590 3712 raft_consensus.cc:787] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20fe6cb0199458c8e038b7230fbd4d2, State: Initialized, Role: FOLLOWER
I20261120 03:03:40.154253 3712 consensus_queue.cc:261] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" member_type: VOTER }
I20261120 03:03:40.154816 3712 raft_consensus.cc:407] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:40.155122 3712 raft_consensus.cc:534] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:40.155423 3712 raft_consensus.cc:3296] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:40.156715 3712 raft_consensus.cc:556] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" member_type: VOTER }
I20261120 03:03:40.160808 3712 leader_election.cc:304] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a20fe6cb0199458c8e038b7230fbd4d2; no voters:
I20261120 03:03:40.161425 3712 leader_election.cc:290] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:40.161507 3715 raft_consensus.cc:3024] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:40.162766 3715 raft_consensus.cc:744] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 1 LEADER]: Becoming Leader. State: Replica: a20fe6cb0199458c8e038b7230fbd4d2, State: Running, Role: LEADER
I20261120 03:03:40.163530 3715 consensus_queue.cc:238] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" member_type: VOTER }
I20261120 03:03:40.165468 3712 sys_catalog.cc:565] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:40.166340 3717 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a20fe6cb0199458c8e038b7230fbd4d2. Latest consensus state: current_term: 1 leader_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" member_type: VOTER } }
I20261120 03:03:40.166312 3716 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" member_type: VOTER } }
I20261120 03:03:40.166976 3716 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:40.166953 3717 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:40.168705 3719 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:40.177263 3719 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:40.189292 3719 catalog_manager.cc:1398] Generated new cluster ID: a71f8f1cb0f0458c886cfa8b99499885
I20261120 03:03:40.189536 3719 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:40.195376 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:40.212721 3719 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:40.213783 3719 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:40.227721 3719 catalog_manager.cc:6237] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2: Generated new TSK 0
I20261120 03:03:40.228312 3719 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:40.261360 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:40.267423 3733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.269688 3734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.272063 3736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:40.272440 385 server_base.cc:1061] running on GCE node
I20261120 03:03:40.273270 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:40.273483 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:40.273661 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143820273644 us; error 0 us; skew 500 ppm
I20261120 03:03:40.274350 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:40.277109 385 webserver.cc:577] Webserver started at http://127.0.96.65:40217/ using document root <none> and password file <none>
I20261120 03:03:40.277555 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:40.277722 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:40.277981 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:40.278906 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "dd8b334dc99d45409536cffa87977dad"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.283298 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20261120 03:03:40.286818 3741 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:40.287652 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20261120 03:03:40.287907 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "dd8b334dc99d45409536cffa87977dad"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.288173 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:40.305012 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:40.305881 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:40.313352 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:40.313606 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:40.313922 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:40.314111 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:40.391618 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:43037
I20261120 03:03:40.391712 3811 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:43037 every 8 connection(s)
I20261120 03:03:40.396592 3812 heartbeater.cc:345] Connected to a master server at 127.0.96.126:35375
I20261120 03:03:40.396984 3812 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:40.397778 3812 heartbeater.cc:508] Master 127.0.96.126:35375 requested a full tablet report, sending...
I20261120 03:03:40.399880 3677 ts_manager.cc:210] Registered new tserver with Master: dd8b334dc99d45409536cffa87977dad (127.0.96.65:43037)
I20261120 03:03:40.400933 385 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.005905478s
I20261120 03:03:40.402272 3677 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33544
I20261120 03:03:40.433107 3774 tablet_service.cc:1495] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261120 03:03:40.434216 3774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:40.444790 3822 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: Bootstrap starting.
I20261120 03:03:40.449271 3822 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:40.453147 3822 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: No bootstrap required, opened a new log
I20261120 03:03:40.453522 3822 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20261120 03:03:40.455349 3822 raft_consensus.cc:366] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "dd8b334dc99d45409536cffa87977dad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43037 } }
I20261120 03:03:40.455751 3822 raft_consensus.cc:392] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:40.455981 3822 raft_consensus.cc:787] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8b334dc99d45409536cffa87977dad, State: Initialized, Role: FOLLOWER
I20261120 03:03:40.456599 3822 consensus_queue.cc:261] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "dd8b334dc99d45409536cffa87977dad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43037 } }
I20261120 03:03:40.457053 3822 raft_consensus.cc:407] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:40.457298 3822 raft_consensus.cc:534] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:40.457556 3822 raft_consensus.cc:3296] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:40.458889 3822 raft_consensus.cc:556] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "dd8b334dc99d45409536cffa87977dad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43037 } }
I20261120 03:03:40.463342 3822 leader_election.cc:304] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd8b334dc99d45409536cffa87977dad; no voters:
I20261120 03:03:40.463927 3822 leader_election.cc:290] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:40.464041 3824 raft_consensus.cc:3024] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:40.465787 3812 heartbeater.cc:500] Master 127.0.96.126:35375 was elected leader, sending a full tablet report...
I20261120 03:03:40.466665 3822 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
I20261120 03:03:40.467075 3824 raft_consensus.cc:744] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 1 LEADER]: Becoming Leader. State: Replica: dd8b334dc99d45409536cffa87977dad, State: Running, Role: LEADER
I20261120 03:03:40.467767 3824 consensus_queue.cc:238] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "dd8b334dc99d45409536cffa87977dad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43037 } }
I20261120 03:03:40.471000 3826 tablet_replica.cc:442] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: TxnStatusTablet state changed. Reason: New leader dd8b334dc99d45409536cffa87977dad. Latest consensus state: current_term: 1 leader_uuid: "dd8b334dc99d45409536cffa87977dad" committed_config { opid_index: -1 peers { permanent_uuid: "dd8b334dc99d45409536cffa87977dad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43037 } } }
I20261120 03:03:40.471580 3826 tablet_replica.cc:445] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: This TxnStatusTablet replica's current role is: LEADER
W20261120 03:03:40.473856 3677 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:40.477933 3825 tablet_replica.cc:442] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd8b334dc99d45409536cffa87977dad" committed_config { opid_index: -1 peers { permanent_uuid: "dd8b334dc99d45409536cffa87977dad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43037 } } }
I20261120 03:03:40.478482 3825 tablet_replica.cc:445] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:40.480010 3829 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261120 03:03:40.480456 3829 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20261120 03:03:40.482400 3677 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
W20261120 03:03:40.527911 3783 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20261120 03:03:40.528252 3783 txn_status_manager.cc:244] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20261120 03:03:40.528581 3783 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20261120 03:03:40.538007 3783 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20261120 03:03:40.572774 3783 txn_status_manager.cc:244] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20261120 03:03:40.573120 3783 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20261120 03:03:40.580839 3783 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20261120 03:03:40.590636 3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:43762
W20261120 03:03:40.592785 3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:43762
W20261120 03:03:40.594630 3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:43762
W20261120 03:03:40.596457 3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:43762
W20261120 03:03:40.598248 3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:43762
W20261120 03:03:40.600021 3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:43762
W20261120 03:03:40.609162 3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:43774
W20261120 03:03:40.610951 3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:43774
W20261120 03:03:40.612811 3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:43774
W20261120 03:03:40.614547 3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:43774
W20261120 03:03:40.616415 3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:43774
W20261120 03:03:40.618290 3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:43774
I20261120 03:03:40.621930 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:40.644902 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:40.645676 385 tablet_replica.cc:333] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: stopping tablet replica
I20261120 03:03:40.646243 385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:40.646761 385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:40.660984 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:40.672039 385 master.cc:585] Master@127.0.96.126:35375 shutting down...
I20261120 03:03:40.692554 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:40.693075 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:40.693516 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2: stopping tablet replica
I20261120 03:03:40.702705 385 master.cc:607] Master@127.0.96.126:35375 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (711 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20261120 03:03:40.734160 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:41445
I20261120 03:03:40.735435 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:40.740741 3837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.741715 3838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.744571 3840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:40.745334 385 server_base.cc:1061] running on GCE node
I20261120 03:03:40.746071 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:40.746281 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:40.746438 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143820746424 us; error 0 us; skew 500 ppm
I20261120 03:03:40.747010 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:40.749281 385 webserver.cc:577] Webserver started at http://127.0.96.126:42135/ using document root <none> and password file <none>
I20261120 03:03:40.749832 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:40.750028 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:40.750314 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:40.751595 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "ff5af720f1ad4794b00b97e9e46d90da"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.757112 385 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20261120 03:03:40.760917 3845 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:40.761646 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20261120 03:03:40.761932 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "ff5af720f1ad4794b00b97e9e46d90da"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.762225 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:40.791435 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:40.792690 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:40.835601 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:41445
I20261120 03:03:40.835657 3896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:41445 every 8 connection(s)
I20261120 03:03:40.840356 3897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:40.850013 3897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da: Bootstrap starting.
I20261120 03:03:40.854591 3897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:40.858253 3897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da: No bootstrap required, opened a new log
I20261120 03:03:40.860522 3897 raft_consensus.cc:366] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5af720f1ad4794b00b97e9e46d90da" member_type: VOTER }
I20261120 03:03:40.860880 3897 raft_consensus.cc:392] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:40.861083 3897 raft_consensus.cc:787] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5af720f1ad4794b00b97e9e46d90da, State: Initialized, Role: FOLLOWER
I20261120 03:03:40.861697 3897 consensus_queue.cc:261] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5af720f1ad4794b00b97e9e46d90da" member_type: VOTER }
I20261120 03:03:40.862141 3897 raft_consensus.cc:407] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:40.862380 3897 raft_consensus.cc:534] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:40.862653 3897 raft_consensus.cc:3296] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:40.863829 3897 raft_consensus.cc:556] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5af720f1ad4794b00b97e9e46d90da" member_type: VOTER }
I20261120 03:03:40.869557 3897 leader_election.cc:304] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff5af720f1ad4794b00b97e9e46d90da; no voters:
I20261120 03:03:40.870023 3897 leader_election.cc:290] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:40.870152 3900 raft_consensus.cc:3024] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:40.873136 3900 raft_consensus.cc:744] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 1 LEADER]: Becoming Leader. State: Replica: ff5af720f1ad4794b00b97e9e46d90da, State: Running, Role: LEADER
I20261120 03:03:40.873337 3897 sys_catalog.cc:565] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:40.873766 3900 consensus_queue.cc:238] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5af720f1ad4794b00b97e9e46d90da" member_type: VOTER }
I20261120 03:03:40.878285 3901 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff5af720f1ad4794b00b97e9e46d90da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5af720f1ad4794b00b97e9e46d90da" member_type: VOTER } }
I20261120 03:03:40.879467 3902 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff5af720f1ad4794b00b97e9e46d90da. Latest consensus state: current_term: 1 leader_uuid: "ff5af720f1ad4794b00b97e9e46d90da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5af720f1ad4794b00b97e9e46d90da" member_type: VOTER } }
I20261120 03:03:40.880240 3902 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:40.880100 3901 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:40.889783 3906 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:40.896148 3906 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:40.898020 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:40.905746 3906 catalog_manager.cc:1398] Generated new cluster ID: c3d34109b22143c69ca30bfd3b28dbfd
I20261120 03:03:40.906001 3906 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:40.925205 3906 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:40.926249 3906 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:40.937126 3906 catalog_manager.cc:6237] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da: Generated new TSK 0
I20261120 03:03:40.937680 3906 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:40.964715 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:40.970655 3918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.971565 3919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.974201 3921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:40.974577 385 server_base.cc:1061] running on GCE node
I20261120 03:03:40.975626 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:40.975873 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:40.976063 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143820976046 us; error 0 us; skew 500 ppm
I20261120 03:03:40.976656 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:40.978881 385 webserver.cc:577] Webserver started at http://127.0.96.65:33811/ using document root <none> and password file <none>
I20261120 03:03:40.979331 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:40.979532 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:40.979795 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:40.980865 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "2a5e1457634e4ac09477344f614be8ad"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.985024 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20261120 03:03:40.987910 3926 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:40.988605 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261120 03:03:40.988885 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "2a5e1457634e4ac09477344f614be8ad"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.989163 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:40.998661 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:40.999577 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:41.004293 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:41.004714 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:41.005105 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:41.005369 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:41.049824 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:44759
I20261120 03:03:41.049921 3996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:44759 every 8 connection(s)
I20261120 03:03:41.054438 3997 heartbeater.cc:345] Connected to a master server at 127.0.96.126:41445
I20261120 03:03:41.054793 3997 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:41.055516 3997 heartbeater.cc:508] Master 127.0.96.126:41445 requested a full tablet report, sending...
I20261120 03:03:41.057473 3862 ts_manager.cc:210] Registered new tserver with Master: 2a5e1457634e4ac09477344f614be8ad (127.0.96.65:44759)
I20261120 03:03:41.057881 385 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004832045s
I20261120 03:03:41.059098 3862 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55114
I20261120 03:03:41.086515 3962 tablet_service.cc:1495] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261120 03:03:41.087561 3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:41.097600 4007 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: Bootstrap starting.
I20261120 03:03:41.101473 4007 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:41.104816 4007 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: No bootstrap required, opened a new log
I20261120 03:03:41.105176 4007 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20261120 03:03:41.106896 4007 raft_consensus.cc:366] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.107267 4007 raft_consensus.cc:392] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:41.107491 4007 raft_consensus.cc:787] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5e1457634e4ac09477344f614be8ad, State: Initialized, Role: FOLLOWER
I20261120 03:03:41.107956 4007 consensus_queue.cc:261] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.108381 4007 raft_consensus.cc:407] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:41.108585 4007 raft_consensus.cc:534] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:41.108831 4007 raft_consensus.cc:3296] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:41.109830 4007 raft_consensus.cc:556] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.114051 4007 leader_election.cc:304] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a5e1457634e4ac09477344f614be8ad; no voters:
I20261120 03:03:41.114545 4007 leader_election.cc:290] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:41.114624 4009 raft_consensus.cc:3024] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:41.115950 4009 raft_consensus.cc:744] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 1 LEADER]: Becoming Leader. State: Replica: 2a5e1457634e4ac09477344f614be8ad, State: Running, Role: LEADER
I20261120 03:03:41.116426 3997 heartbeater.cc:500] Master 127.0.96.126:41445 was elected leader, sending a full tablet report...
I20261120 03:03:41.116847 4007 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20261120 03:03:41.116638 4009 consensus_queue.cc:238] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.119297 4010 tablet_replica.cc:442] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a5e1457634e4ac09477344f614be8ad" committed_config { opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } } }
I20261120 03:03:41.119836 4010 tablet_replica.cc:445] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:41.119666 4011 tablet_replica.cc:442] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: TxnStatusTablet state changed. Reason: New leader 2a5e1457634e4ac09477344f614be8ad. Latest consensus state: current_term: 1 leader_uuid: "2a5e1457634e4ac09477344f614be8ad" committed_config { opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } } }
I20261120 03:03:41.120205 4011 tablet_replica.cc:445] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: This TxnStatusTablet replica's current role is: LEADER
W20261120 03:03:41.122058 3862 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:41.122148 4013 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261120 03:03:41.122555 4013 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20261120 03:03:41.125686 3862 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:41.142879 3962 tablet_service.cc:1495] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261120 03:03:41.143806 3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:41.153038 4007 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad: Bootstrap starting.
I20261120 03:03:41.157117 4007 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:41.160583 4007 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad: No bootstrap required, opened a new log
I20261120 03:03:41.160971 4007 ts_tablet_manager.cc:1449] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20261120 03:03:41.162745 4007 raft_consensus.cc:366] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.163146 4007 raft_consensus.cc:392] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:41.163371 4007 raft_consensus.cc:787] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5e1457634e4ac09477344f614be8ad, State: Initialized, Role: FOLLOWER
I20261120 03:03:41.163842 4007 consensus_queue.cc:261] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.164400 4007 raft_consensus.cc:407] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:41.164644 4007 raft_consensus.cc:534] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:41.164911 4007 raft_consensus.cc:3296] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:41.165422 4007 raft_consensus.cc:556] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.169600 4007 leader_election.cc:304] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a5e1457634e4ac09477344f614be8ad; no voters:
I20261120 03:03:41.170068 4007 leader_election.cc:290] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:41.170171 4009 raft_consensus.cc:3024] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:41.170687 4009 raft_consensus.cc:744] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 1 LEADER]: Becoming Leader. State: Replica: 2a5e1457634e4ac09477344f614be8ad, State: Running, Role: LEADER
I20261120 03:03:41.171363 4009 consensus_queue.cc:238] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.171980 4007 ts_tablet_manager.cc:1480] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
W20261120 03:03:41.176295 3862 catalog_manager.cc:5689] Ignoring report from unknown tablet 22222222222222222222222222222222
W20261120 03:03:41.179756 3862 catalog_manager.cc:5689] Ignoring report from unknown tablet 22222222222222222222222222222222
I20261120 03:03:41.200241 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:41.218407 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:41.219049 385 tablet_replica.cc:333] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad: stopping tablet replica
I20261120 03:03:41.219542 385 raft_consensus.cc:2440] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:41.219938 385 raft_consensus.cc:2469] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:41.222157 385 tablet_replica.cc:333] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: stopping tablet replica
I20261120 03:03:41.222592 385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:41.222961 385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:41.233364 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:41.243028 385 master.cc:585] Master@127.0.96.126:41445 shutting down...
I20261120 03:03:41.256150 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:41.256629 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:41.257108 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da: stopping tablet replica
I20261120 03:03:41.275498 385 master.cc:607] Master@127.0.96.126:41445 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (565 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (3250 ms total)
[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/0
I20261120 03:03:41.298225 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:33659
I20261120 03:03:41.299171 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:41.303613 4023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:41.304847 4024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:41.306308 385 server_base.cc:1061] running on GCE node
W20261120 03:03:41.307052 4026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:41.307755 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:41.307962 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:41.308156 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143821308099 us; error 0 us; skew 500 ppm
I20261120 03:03:41.308710 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:41.310799 385 webserver.cc:577] Webserver started at http://127.0.96.126:33651/ using document root <none> and password file <none>
I20261120 03:03:41.311371 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:41.311558 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:41.311825 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:41.312868 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "7d9d0a04125446849a404f868bc1cf29"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.317116 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20261120 03:03:41.320200 4031 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:41.320843 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 03:03:41.321163 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "7d9d0a04125446849a404f868bc1cf29"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.321416 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:41.332790 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:41.333626 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:41.365540 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:33659
I20261120 03:03:41.365624 4082 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:33659 every 8 connection(s)
I20261120 03:03:41.369102 4083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:41.378253 4083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29: Bootstrap starting.
I20261120 03:03:41.382076 4083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:41.385432 4083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29: No bootstrap required, opened a new log
I20261120 03:03:41.387146 4083 raft_consensus.cc:366] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9d0a04125446849a404f868bc1cf29" member_type: VOTER }
I20261120 03:03:41.387478 4083 raft_consensus.cc:392] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:41.387645 4083 raft_consensus.cc:787] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9d0a04125446849a404f868bc1cf29, State: Initialized, Role: FOLLOWER
I20261120 03:03:41.388090 4083 consensus_queue.cc:261] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9d0a04125446849a404f868bc1cf29" member_type: VOTER }
I20261120 03:03:41.388540 4083 raft_consensus.cc:407] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:41.388722 4083 raft_consensus.cc:534] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:41.388914 4083 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:41.389823 4083 raft_consensus.cc:556] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9d0a04125446849a404f868bc1cf29" member_type: VOTER }
I20261120 03:03:41.394058 4083 leader_election.cc:304] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d9d0a04125446849a404f868bc1cf29; no voters:
I20261120 03:03:41.394572 4083 leader_election.cc:290] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:41.394675 4086 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:41.396109 4086 raft_consensus.cc:744] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 1 LEADER]: Becoming Leader. State: Replica: 7d9d0a04125446849a404f868bc1cf29, State: Running, Role: LEADER
I20261120 03:03:41.396765 4086 consensus_queue.cc:238] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9d0a04125446849a404f868bc1cf29" member_type: VOTER }
I20261120 03:03:41.397466 4083 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:41.398994 4087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d9d0a04125446849a404f868bc1cf29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9d0a04125446849a404f868bc1cf29" member_type: VOTER } }
I20261120 03:03:41.400556 4087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:41.399967 4088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d9d0a04125446849a404f868bc1cf29. Latest consensus state: current_term: 1 leader_uuid: "7d9d0a04125446849a404f868bc1cf29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9d0a04125446849a404f868bc1cf29" member_type: VOTER } }
I20261120 03:03:41.401556 4088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:41.404686 4093 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:41.408788 4093 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:41.411461 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:41.416814 4093 catalog_manager.cc:1398] Generated new cluster ID: cd3a48158fbc443ca1058733e20dfd72
I20261120 03:03:41.417060 4093 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:41.437521 4093 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:41.438637 4093 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:41.452047 4093 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29: Generated new TSK 0
I20261120 03:03:41.452585 4093 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:41.477620 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:41.483075 4104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:41.484180 4105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:41.486274 4107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:41.486425 385 server_base.cc:1061] running on GCE node
I20261120 03:03:41.487218 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:41.487444 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:41.487633 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143821487599 us; error 0 us; skew 500 ppm
I20261120 03:03:41.488238 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:41.490120 385 webserver.cc:577] Webserver started at http://127.0.96.65:35397/ using document root <none> and password file <none>
I20261120 03:03:41.490566 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:41.490819 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:41.491082 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:41.492120 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "5686819a367f4567b741ac52b51c69a3"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.496287 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20261120 03:03:41.499235 4112 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:41.499929 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 03:03:41.500211 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "5686819a367f4567b741ac52b51c69a3"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.500460 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:41.528260 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:41.529086 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:41.530413 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:41.532538 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:41.532917 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:41.533177 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:41.533346 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:41.567986 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:37193
I20261120 03:03:41.568096 4174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:37193 every 8 connection(s)
I20261120 03:03:41.572611 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:41.578867 4179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:41.579675 4180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:41.582686 4182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:41.583205 385 server_base.cc:1061] running on GCE node
I20261120 03:03:41.583747 4175 heartbeater.cc:345] Connected to a master server at 127.0.96.126:33659
I20261120 03:03:41.584174 4175 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:41.584396 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:41.584645 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:41.584848 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143821584830 us; error 0 us; skew 500 ppm
I20261120 03:03:41.584931 4175 heartbeater.cc:508] Master 127.0.96.126:33659 requested a full tablet report, sending...
I20261120 03:03:41.585547 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:41.587034 4048 ts_manager.cc:210] Registered new tserver with Master: 5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193)
I20261120 03:03:41.588285 385 webserver.cc:577] Webserver started at http://127.0.96.66:42431/ using document root <none> and password file <none>
I20261120 03:03:41.588747 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:41.588922 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:41.588912 4048 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44338
I20261120 03:03:41.589334 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:41.590415 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "e554bd281f3844c098fb147463c08a84"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.594368 385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20261120 03:03:41.597323 4187 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:41.598014 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 03:03:41.598266 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "e554bd281f3844c098fb147463c08a84"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.598501 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:41.606246 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:41.607110 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:41.608439 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:41.610357 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:41.610548 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:41.610761 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:41.610935 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:41.644881 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:45765
I20261120 03:03:41.644966 4249 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:45765 every 8 connection(s)
I20261120 03:03:41.649176 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:41.656430 4253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:41.656517 4254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:41.659888 4250 heartbeater.cc:345] Connected to a master server at 127.0.96.126:33659
I20261120 03:03:41.660374 4250 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:41.660789 385 server_base.cc:1061] running on GCE node
W20261120 03:03:41.660785 4256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:41.661147 4250 heartbeater.cc:508] Master 127.0.96.126:33659 requested a full tablet report, sending...
I20261120 03:03:41.661780 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:41.662009 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:41.662176 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143821662163 us; error 0 us; skew 500 ppm
I20261120 03:03:41.662904 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:41.663110 4048 ts_manager.cc:210] Registered new tserver with Master: e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765)
I20261120 03:03:41.664584 4048 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44344
I20261120 03:03:41.665344 385 webserver.cc:577] Webserver started at http://127.0.96.67:41609/ using document root <none> and password file <none>
I20261120 03:03:41.665824 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:41.666026 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:41.666280 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:41.667299 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-2-root/instance:
uuid: "c94544a8d7604d15b3a451f3657b06a7"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.671358 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20261120 03:03:41.674188 4261 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:41.674993 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20261120 03:03:41.675215 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-2-root
uuid: "c94544a8d7604d15b3a451f3657b06a7"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.675431 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:41.685007 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:41.685797 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:41.686866 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:41.688683 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:41.688843 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:41.689095 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:41.689227 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:41.723927 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.67:45369
I20261120 03:03:41.724020 4323 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.67:45369 every 8 connection(s)
I20261120 03:03:41.736411 4324 heartbeater.cc:345] Connected to a master server at 127.0.96.126:33659
I20261120 03:03:41.736753 4324 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:41.737376 4324 heartbeater.cc:508] Master 127.0.96.126:33659 requested a full tablet report, sending...
I20261120 03:03:41.738946 4048 ts_manager.cc:210] Registered new tserver with Master: c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369)
I20261120 03:03:41.739969 385 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013123899s
I20261120 03:03:41.740010 4048 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44350
I20261120 03:03:41.776002 4048 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44368:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261120 03:03:41.819103 4289 tablet_service.cc:1495] Processing CreateTablet for tablet 961865cf1efc4c71a4a12606b68f48d5 (DEFAULT_TABLE table=test-workload [id=74fbc5a4629d4a4aa0d813b8d2120ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:41.820379 4289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961865cf1efc4c71a4a12606b68f48d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:41.821167 4140 tablet_service.cc:1495] Processing CreateTablet for tablet 961865cf1efc4c71a4a12606b68f48d5 (DEFAULT_TABLE table=test-workload [id=74fbc5a4629d4a4aa0d813b8d2120ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:41.822368 4140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961865cf1efc4c71a4a12606b68f48d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:41.830118 4215 tablet_service.cc:1495] Processing CreateTablet for tablet 961865cf1efc4c71a4a12606b68f48d5 (DEFAULT_TABLE table=test-workload [id=74fbc5a4629d4a4aa0d813b8d2120ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:41.831264 4215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961865cf1efc4c71a4a12606b68f48d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:41.836215 4344 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Bootstrap starting.
I20261120 03:03:41.841174 4345 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3: Bootstrap starting.
I20261120 03:03:41.843219 4344 tablet_bootstrap.cc:654] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:41.847738 4345 tablet_bootstrap.cc:654] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:41.848511 4346 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: Bootstrap starting.
I20261120 03:03:41.851770 4344 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: No bootstrap required, opened a new log
I20261120 03:03:41.852245 4344 ts_tablet_manager.cc:1449] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20261120 03:03:41.852553 4345 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3: No bootstrap required, opened a new log
I20261120 03:03:41.852994 4345 ts_tablet_manager.cc:1449] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20261120 03:03:41.854224 4346 tablet_bootstrap.cc:654] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:41.855042 4344 raft_consensus.cc:366] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.855865 4344 raft_consensus.cc:392] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:41.855720 4345 raft_consensus.cc:366] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.856241 4344 raft_consensus.cc:787] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94544a8d7604d15b3a451f3657b06a7, State: Initialized, Role: FOLLOWER
I20261120 03:03:41.856462 4345 raft_consensus.cc:392] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:41.856789 4345 raft_consensus.cc:787] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5686819a367f4567b741ac52b51c69a3, State: Initialized, Role: FOLLOWER
I20261120 03:03:41.857089 4344 consensus_queue.cc:261] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.857481 4345 consensus_queue.cc:261] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.860028 4175 heartbeater.cc:500] Master 127.0.96.126:33659 was elected leader, sending a full tablet report...
I20261120 03:03:41.860716 4345 ts_tablet_manager.cc:1480] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20261120 03:03:41.862874 4324 heartbeater.cc:500] Master 127.0.96.126:33659 was elected leader, sending a full tablet report...
I20261120 03:03:41.863874 4344 ts_tablet_manager.cc:1480] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Time spent starting tablet: real 0.011s user 0.002s sys 0.008s
I20261120 03:03:41.864183 4346 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: No bootstrap required, opened a new log
I20261120 03:03:41.864670 4346 ts_tablet_manager.cc:1449] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20261120 03:03:41.867216 4346 raft_consensus.cc:366] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.867726 4346 raft_consensus.cc:392] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:41.867964 4346 raft_consensus.cc:787] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e554bd281f3844c098fb147463c08a84, State: Initialized, Role: FOLLOWER
I20261120 03:03:41.868701 4346 consensus_queue.cc:261] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.871793 4346 ts_tablet_manager.cc:1480] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20261120 03:03:41.872277 4250 heartbeater.cc:500] Master 127.0.96.126:33659 was elected leader, sending a full tablet report...
I20261120 03:03:41.923336 4350 raft_consensus.cc:534] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:41.923703 4350 raft_consensus.cc:556] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.925637 4350 leader_election.cc:290] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193), e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765)
I20261120 03:03:41.934147 4150 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "c94544a8d7604d15b3a451f3657b06a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5686819a367f4567b741ac52b51c69a3" is_pre_election: true
I20261120 03:03:41.934981 4150 raft_consensus.cc:2062] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate c94544a8d7604d15b3a451f3657b06a7 in term 0
I20261120 03:03:41.935662 4225 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "c94544a8d7604d15b3a451f3657b06a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e554bd281f3844c098fb147463c08a84" is_pre_election: true
I20261120 03:03:41.935972 4264 leader_election.cc:304] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5686819a367f4567b741ac52b51c69a3, c94544a8d7604d15b3a451f3657b06a7; no voters:
I20261120 03:03:41.936336 4225 raft_consensus.cc:2062] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate c94544a8d7604d15b3a451f3657b06a7 in term 0
I20261120 03:03:41.936715 4350 raft_consensus.cc:3024] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:41.937031 4350 raft_consensus.cc:534] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:41.937307 4350 raft_consensus.cc:3296] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:41.938187 4350 raft_consensus.cc:556] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.942989 4350 leader_election.cc:290] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [CANDIDATE]: Term 1 election: Requested vote from peers 5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193), e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765)
I20261120 03:03:41.943624 4150 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "c94544a8d7604d15b3a451f3657b06a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5686819a367f4567b741ac52b51c69a3"
I20261120 03:03:41.943830 4225 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "c94544a8d7604d15b3a451f3657b06a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e554bd281f3844c098fb147463c08a84"
I20261120 03:03:41.944176 4150 raft_consensus.cc:3296] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:41.944373 4225 raft_consensus.cc:3296] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:41.945245 4150 raft_consensus.cc:2062] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate c94544a8d7604d15b3a451f3657b06a7 in term 1
I20261120 03:03:41.945323 4225 raft_consensus.cc:2062] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate c94544a8d7604d15b3a451f3657b06a7 in term 1
I20261120 03:03:41.952020 4264 leader_election.cc:304] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c94544a8d7604d15b3a451f3657b06a7, e554bd281f3844c098fb147463c08a84; no voters:
I20261120 03:03:41.952759 4350 raft_consensus.cc:3024] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:41.953161 4350 raft_consensus.cc:744] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 1 LEADER]: Becoming Leader. State: Replica: c94544a8d7604d15b3a451f3657b06a7, State: Running, Role: LEADER
I20261120 03:03:41.953874 4350 consensus_queue.cc:238] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.959921 4354 consensus_queue.cc:1049] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:41.962930 4047 catalog_manager.cc:5856] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67). New cstate: current_term: 1 leader_uuid: "c94544a8d7604d15b3a451f3657b06a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } }
I20261120 03:03:41.966979 4356 consensus_queue.cc:1049] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261120 03:03:42.220103 4140 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:42.222905 4289 tablet_service.cc:1451] Tablet server has 1 leaders and 0 scanners
I20261120 03:03:42.223083 4215 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7d9d0a04125446849a404f868bc1cf29 | 127.0.96.126:33659 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
5686819a367f4567b741ac52b51c69a3 | 127.0.96.65:37193 | HEALTHY | <none> | 0 | 0
c94544a8d7604d15b3a451f3657b06a7 | 127.0.96.67:45369 | HEALTHY | <none> | 1 | 0
e554bd281f3844c098fb147463c08a84 | 127.0.96.66:45765 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20261120 03:03:42.443405 385 log_verifier.cc:126] Checking tablet 961865cf1efc4c71a4a12606b68f48d5
I20261120 03:03:42.502789 385 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 961865cf1efc4c71a4a12606b68f48d5
I20261120 03:03:42.503320 385 tablet_replica.cc:333] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: stopping tablet replica
I20261120 03:03:42.503786 385 raft_consensus.cc:2440] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:42.504597 385 raft_consensus.cc:2469] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:42.700480 4140 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:42.700588 4289 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:42.702387 4215 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7d9d0a04125446849a404f868bc1cf29 | 127.0.96.126:33659 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
5686819a367f4567b741ac52b51c69a3 | 127.0.96.65:37193 | HEALTHY | <none> | 0 | 0
c94544a8d7604d15b3a451f3657b06a7 | 127.0.96.67:45369 | HEALTHY | <none> | 0 | 0
e554bd281f3844c098fb147463c08a84 | 127.0.96.66:45765 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 961865cf1efc4c71a4a12606b68f48d5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193): RUNNING
e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765): RUNNING
All reported replicas are:
A = c94544a8d7604d15b3a451f3657b06a7
B = 5686819a367f4567b741ac52b51c69a3
C = e554bd281f3844c098fb147463c08a84
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | A* B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20261120 03:03:42.896771 385 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261120 03:03:43.242240 4140 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:43.242638 4289 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:43.246492 4215 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7d9d0a04125446849a404f868bc1cf29 | 127.0.96.126:33659 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
5686819a367f4567b741ac52b51c69a3 | 127.0.96.65:37193 | HEALTHY | <none> | 0 | 0
c94544a8d7604d15b3a451f3657b06a7 | 127.0.96.67:45369 | HEALTHY | <none> | 0 | 0
e554bd281f3844c098fb147463c08a84 | 127.0.96.66:45765 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 961865cf1efc4c71a4a12606b68f48d5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193): RUNNING
e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765): RUNNING
All reported replicas are:
A = c94544a8d7604d15b3a451f3657b06a7
B = 5686819a367f4567b741ac52b51c69a3
C = e554bd281f3844c098fb147463c08a84
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | A* B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20261120 03:03:43.440987 385 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261120 03:03:43.866423 4140 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:43.873127 4289 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:43.874809 4215 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:43.881464 4463 raft_consensus.cc:534] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c94544a8d7604d15b3a451f3657b06a7)
I20261120 03:03:43.882016 4463 raft_consensus.cc:556] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:43.932756 4465 raft_consensus.cc:534] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c94544a8d7604d15b3a451f3657b06a7)
I20261120 03:03:43.933027 4463 leader_election.cc:290] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369), 5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193)
I20261120 03:03:43.933378 4465 raft_consensus.cc:556] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:43.954052 4465 leader_election.cc:290] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369), e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765)
I20261120 03:03:43.958839 4149 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "e554bd281f3844c098fb147463c08a84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5686819a367f4567b741ac52b51c69a3" is_pre_election: true
I20261120 03:03:43.961010 4149 raft_consensus.cc:2062] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate e554bd281f3844c098fb147463c08a84 in term 1
I20261120 03:03:43.962296 4190 leader_election.cc:304] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5686819a367f4567b741ac52b51c69a3, e554bd281f3844c098fb147463c08a84; no voters:
I20261120 03:03:43.963137 4463 raft_consensus.cc:3024] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:43.963522 4463 raft_consensus.cc:534] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c94544a8d7604d15b3a451f3657b06a7)
I20261120 03:03:43.963860 4463 raft_consensus.cc:3296] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:43.972311 4463 raft_consensus.cc:556] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:43.981005 4225 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "5686819a367f4567b741ac52b51c69a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e554bd281f3844c098fb147463c08a84" is_pre_election: true
I20261120 03:03:43.981769 4225 raft_consensus.cc:2628] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5686819a367f4567b741ac52b51c69a3 in current term 2: Already voted for candidate e554bd281f3844c098fb147463c08a84 in this term.
I20261120 03:03:43.982640 4299 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "5686819a367f4567b741ac52b51c69a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c94544a8d7604d15b3a451f3657b06a7" is_pre_election: true
I20261120 03:03:43.984457 4299 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "e554bd281f3844c098fb147463c08a84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c94544a8d7604d15b3a451f3657b06a7" is_pre_election: true
I20261120 03:03:43.989393 4463 leader_election.cc:290] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [CANDIDATE]: Term 2 election: Requested vote from peers c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369), 5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193)
I20261120 03:03:43.989914 4149 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "e554bd281f3844c098fb147463c08a84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5686819a367f4567b741ac52b51c69a3"
W20261120 03:03:43.990407 4189 leader_election.cc:343] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): Illegal state: cannot vote while shut down
I20261120 03:03:43.992056 4299 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "e554bd281f3844c098fb147463c08a84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c94544a8d7604d15b3a451f3657b06a7"
I20261120 03:03:43.990428 4149 raft_consensus.cc:3296] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 1 FOLLOWER]: Advancing to term 2
W20261120 03:03:43.993280 4189 leader_election.cc:343] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): Illegal state: cannot vote while shut down
I20261120 03:03:43.994136 4149 raft_consensus.cc:2062] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate e554bd281f3844c098fb147463c08a84 in term 2
W20261120 03:03:43.994253 4114 leader_election.cc:343] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): Illegal state: cannot vote while shut down
I20261120 03:03:43.999852 4114 leader_election.cc:304] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5686819a367f4567b741ac52b51c69a3; no voters: c94544a8d7604d15b3a451f3657b06a7, e554bd281f3844c098fb147463c08a84
I20261120 03:03:44.002434 4465 raft_consensus.cc:2966] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261120 03:03:44.003104 4190 leader_election.cc:304] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5686819a367f4567b741ac52b51c69a3, e554bd281f3844c098fb147463c08a84; no voters: c94544a8d7604d15b3a451f3657b06a7
I20261120 03:03:44.004276 4463 raft_consensus.cc:3024] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:44.004753 4463 raft_consensus.cc:744] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 LEADER]: Becoming Leader. State: Replica: e554bd281f3844c098fb147463c08a84, State: Running, Role: LEADER
I20261120 03:03:44.005555 4463 consensus_queue.cc:238] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:44.019018 4149 raft_consensus.cc:1338] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Refusing update from remote peer e554bd281f3844c098fb147463c08a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261120 03:03:44.020713 4470 consensus_queue.cc:1049] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20261120 03:03:44.020612 4048 catalog_manager.cc:5856] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 reported cstate change: term changed from 1 to 2, leader changed from c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) to e554bd281f3844c098fb147463c08a84 (127.0.96.66). New cstate: current_term: 2 leader_uuid: "e554bd281f3844c098fb147463c08a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } }
W20261120 03:03:44.028003 4189 consensus_peers.cc:597] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 -> Peer c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): Couldn't send request to peer c94544a8d7604d15b3a451f3657b06a7. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7d9d0a04125446849a404f868bc1cf29 | 127.0.96.126:33659 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
5686819a367f4567b741ac52b51c69a3 | 127.0.96.65:37193 | HEALTHY | <none> | 0 | 0
c94544a8d7604d15b3a451f3657b06a7 | 127.0.96.67:45369 | HEALTHY | <none> | 0 | 0
e554bd281f3844c098fb147463c08a84 | 127.0.96.66:45765 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 961865cf1efc4c71a4a12606b68f48d5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193): RUNNING
e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765): RUNNING
All reported replicas are:
A = c94544a8d7604d15b3a451f3657b06a7
B = 5686819a367f4567b741ac52b51c69a3
C = e554bd281f3844c098fb147463c08a84
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | A* B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20261120 03:03:44.140944 385 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261120 03:03:44.592993 4463 raft_consensus.cc:1070] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: Attempting to remove follower c94544a8d7604d15b3a451f3657b06a7 from the Raft config. Reason: The tablet replica hosted on peer c94544a8d7604d15b3a451f3657b06a7 has failed
I20261120 03:03:44.595585 4463 consensus_queue.cc:238] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:44.598932 4149 raft_consensus.cc:1338] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Refusing update from remote peer e554bd281f3844c098fb147463c08a84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:44.600278 4488 consensus_queue.cc:1049] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:44.605387 4463 raft_consensus.cc:3183] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } }
I20261120 03:03:44.608626 4149 raft_consensus.cc:3183] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } }
I20261120 03:03:44.611158 4046 catalog_manager.cc:5856] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 reported cstate change: config changed from index -1 to 3, VOTER c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) evicted. New cstate: current_term: 2 leader_uuid: "e554bd281f3844c098fb147463c08a84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } }
I20261120 03:03:44.626425 4289 tablet_service.cc:1542] Processing DeleteTablet for tablet 961865cf1efc4c71a4a12606b68f48d5 with delete_type TABLET_DATA_TOMBSTONED (TS c94544a8d7604d15b3a451f3657b06a7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50226
I20261120 03:03:44.628680 4499 ts_tablet_manager.cc:1962] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20261120 03:03:44.628845 4225 consensus_queue.cc:238] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } }
W20261120 03:03:44.634150 4189 consensus_peers.cc:597] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 -> Peer c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): Couldn't send request to peer c94544a8d7604d15b3a451f3657b06a7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:44.634547 4149 raft_consensus.cc:1338] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Refusing update from remote peer e554bd281f3844c098fb147463c08a84: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20261120 03:03:44.635881 4488 consensus_queue.cc:1049] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:44.641222 4463 raft_consensus.cc:3183] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } }
I20261120 03:03:44.642745 4149 raft_consensus.cc:3183] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } }
I20261120 03:03:44.644281 4034 catalog_manager.cc:5263] ChangeConfig:ADD_PEER:VOTER RPC for tablet 961865cf1efc4c71a4a12606b68f48d5 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20261120 03:03:44.645140 4499 ts_tablet_manager.cc:1975] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20261120 03:03:44.645622 4499 log.cc:1199] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Deleting WAL directory at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-2-root/wals/961865cf1efc4c71a4a12606b68f48d5
I20261120 03:03:44.647583 4033 catalog_manager.cc:5081] TS c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): tablet 961865cf1efc4c71a4a12606b68f48d5 (table test-workload [id=74fbc5a4629d4a4aa0d813b8d2120ae4]) successfully deleted
I20261120 03:03:44.649461 4046 catalog_manager.cc:5856] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 reported cstate change: config changed from index 3 to 4, VOTER c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) added. New cstate: current_term: 2 leader_uuid: "e554bd281f3844c098fb147463c08a84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } }
I20261120 03:03:44.720041 4140 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:44.729888 4289 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:44.733116 4215 tablet_service.cc:1451] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7d9d0a04125446849a404f868bc1cf29 | 127.0.96.126:33659 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
5686819a367f4567b741ac52b51c69a3 | 127.0.96.65:37193 | HEALTHY | <none> | 0 | 0
c94544a8d7604d15b3a451f3657b06a7 | 127.0.96.67:45369 | HEALTHY | <none> | 0 | 0
e554bd281f3844c098fb147463c08a84 | 127.0.96.66:45765 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 961865cf1efc4c71a4a12606b68f48d5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193): RUNNING
e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765): RUNNING [LEADER]
c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 5686819a367f4567b741ac52b51c69a3
B = e554bd281f3844c098fb147463c08a84
C = c94544a8d7604d15b3a451f3657b06a7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 4 | Yes
B | A B* C | 2 | 4 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20261120 03:03:44.923781 385 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261120 03:03:45.138525 4514 ts_tablet_manager.cc:979] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Initiating tablet copy from peer e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765)
I20261120 03:03:45.139796 4514 tablet_copy_client.cc:323] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.96.66:45765
I20261120 03:03:45.141235 4235 tablet_copy_service.cc:140] P e554bd281f3844c098fb147463c08a84: Received BeginTabletCopySession request for tablet 961865cf1efc4c71a4a12606b68f48d5 from peer c94544a8d7604d15b3a451f3657b06a7 ({username='slave'} at 127.0.0.1:43716)
I20261120 03:03:45.141659 4235 tablet_copy_service.cc:161] P e554bd281f3844c098fb147463c08a84: Beginning new tablet copy session on tablet 961865cf1efc4c71a4a12606b68f48d5 from peer c94544a8d7604d15b3a451f3657b06a7 at {username='slave'} at 127.0.0.1:43716: session id = c94544a8d7604d15b3a451f3657b06a7-961865cf1efc4c71a4a12606b68f48d5
I20261120 03:03:45.146488 4235 tablet_copy_source_session.cc:215] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: Tablet Copy: opened 0 blocks and 1 log segments
I20261120 03:03:45.148399 4514 ts_tablet_manager.cc:1962] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Deleting tablet data with delete state TABLET_DATA_COPYING
I20261120 03:03:45.158772 4514 ts_tablet_manager.cc:1975] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20261120 03:03:45.159324 4514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961865cf1efc4c71a4a12606b68f48d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:45.164237 4514 tablet_copy_client.cc:806] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: tablet copy: Starting download of 0 data blocks...
I20261120 03:03:45.164649 4514 tablet_copy_client.cc:670] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: tablet copy: Starting download of 1 WAL segments...
I20261120 03:03:45.167639 4514 tablet_copy_client.cc:538] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261120 03:03:45.172911 4514 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Bootstrap starting.
I20261120 03:03:45.188017 4514 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:45.188683 4514 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Bootstrap complete.
I20261120 03:03:45.189182 4514 ts_tablet_manager.cc:1449] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.006s
I20261120 03:03:45.190989 4514 raft_consensus.cc:366] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } }
I20261120 03:03:45.191574 4514 raft_consensus.cc:787] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94544a8d7604d15b3a451f3657b06a7, State: Initialized, Role: FOLLOWER
I20261120 03:03:45.192199 4514 consensus_queue.cc:261] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } }
I20261120 03:03:45.194197 4514 ts_tablet_manager.cc:1480] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20261120 03:03:45.195812 4235 tablet_copy_service.cc:342] P e554bd281f3844c098fb147463c08a84: Request end of tablet copy session c94544a8d7604d15b3a451f3657b06a7-961865cf1efc4c71a4a12606b68f48d5 received from {username='slave'} at 127.0.0.1:43716
I20261120 03:03:45.196193 4235 tablet_copy_service.cc:434] P e554bd281f3844c098fb147463c08a84: ending tablet copy session c94544a8d7604d15b3a451f3657b06a7-961865cf1efc4c71a4a12606b68f48d5 on tablet 961865cf1efc4c71a4a12606b68f48d5 with peer c94544a8d7604d15b3a451f3657b06a7
I20261120 03:03:45.614084 4299 raft_consensus.cc:1279] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4] Dedup: 2.4->[]
I20261120 03:03:45.633736 4289 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:45.638548 4140 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:45.640870 4215 tablet_service.cc:1451] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7d9d0a04125446849a404f868bc1cf29 | 127.0.96.126:33659 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
5686819a367f4567b741ac52b51c69a3 | 127.0.96.65:37193 | HEALTHY | <none> | 0 | 0
c94544a8d7604d15b3a451f3657b06a7 | 127.0.96.67:45369 | HEALTHY | <none> | 0 | 0
e554bd281f3844c098fb147463c08a84 | 127.0.96.66:45765 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20261120 03:03:45.846527 385 log_verifier.cc:126] Checking tablet 961865cf1efc4c71a4a12606b68f48d5
I20261120 03:03:45.896865 385 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 961865cf1efc4c71a4a12606b68f48d5
I20261120 03:03:45.899945 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:45.923861 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:45.924510 385 tablet_replica.cc:333] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3: stopping tablet replica
I20261120 03:03:45.925022 385 raft_consensus.cc:2440] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:45.925514 385 raft_consensus.cc:2469] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:45.944460 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:45.952631 385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:45.969669 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:45.970145 385 tablet_replica.cc:333] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: stopping tablet replica
I20261120 03:03:45.970569 385 raft_consensus.cc:2440] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 LEADER]: Raft consensus shutting down.
I20261120 03:03:45.971345 385 raft_consensus.cc:2469] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:45.989751 385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:45.997961 385 tablet_server.cc:179] TabletServer@127.0.96.67:0 shutting down...
I20261120 03:03:46.013463 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:46.014002 385 tablet_replica.cc:333] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: stopping tablet replica
I20261120 03:03:46.014456 385 raft_consensus.cc:2440] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:46.014849 385 raft_consensus.cc:2469] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:46.033376 385 tablet_server.cc:196] TabletServer@127.0.96.67:0 shutdown complete.
I20261120 03:03:46.041604 385 master.cc:585] Master@127.0.96.126:33659 shutting down...
I20261120 03:03:46.054625 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:46.055089 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:46.055446 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29: stopping tablet replica
I20261120 03:03:46.073941 385 master.cc:607] Master@127.0.96.126:33659 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (4802 ms)
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/1
I20261120 03:03:46.099690 385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:45111
I20261120 03:03:46.100631 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:46.105304 4549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.106446 4550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.107489 4552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.108047 385 server_base.cc:1061] running on GCE node
I20261120 03:03:46.108997 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:46.109228 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:46.109400 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143826109369 us; error 0 us; skew 500 ppm
I20261120 03:03:46.109911 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:46.111814 385 webserver.cc:577] Webserver started at http://127.0.96.126:43999/ using document root <none> and password file <none>
I20261120 03:03:46.112322 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:46.112505 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:46.112787 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:46.113786 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "ab901ce6ae784c0997186711f9a199c2"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.117836 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20261120 03:03:46.120779 4557 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.121517 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 03:03:46.121783 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "ab901ce6ae784c0997186711f9a199c2"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.122025 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:46.137247 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:46.138085 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:46.169865 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:45111
I20261120 03:03:46.169950 4608 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:45111 every 8 connection(s)
I20261120 03:03:46.173592 4609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:46.183063 4609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2: Bootstrap starting.
I20261120 03:03:46.187094 4609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:46.190445 4609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2: No bootstrap required, opened a new log
I20261120 03:03:46.192219 4609 raft_consensus.cc:366] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab901ce6ae784c0997186711f9a199c2" member_type: VOTER }
I20261120 03:03:46.192544 4609 raft_consensus.cc:392] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:46.192713 4609 raft_consensus.cc:787] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab901ce6ae784c0997186711f9a199c2, State: Initialized, Role: FOLLOWER
I20261120 03:03:46.193161 4609 consensus_queue.cc:261] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab901ce6ae784c0997186711f9a199c2" member_type: VOTER }
I20261120 03:03:46.193519 4609 raft_consensus.cc:407] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:46.193704 4609 raft_consensus.cc:534] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:46.193900 4609 raft_consensus.cc:3296] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:46.194837 4609 raft_consensus.cc:556] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab901ce6ae784c0997186711f9a199c2" member_type: VOTER }
I20261120 03:03:46.198951 4609 leader_election.cc:304] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab901ce6ae784c0997186711f9a199c2; no voters:
I20261120 03:03:46.199404 4609 leader_election.cc:290] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 03:03:46.199508 4612 raft_consensus.cc:3024] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:46.200698 4612 raft_consensus.cc:744] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 1 LEADER]: Becoming Leader. State: Replica: ab901ce6ae784c0997186711f9a199c2, State: Running, Role: LEADER
I20261120 03:03:46.201313 4612 consensus_queue.cc:238] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab901ce6ae784c0997186711f9a199c2" member_type: VOTER }
I20261120 03:03:46.202098 4609 sys_catalog.cc:565] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:46.203719 4614 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab901ce6ae784c0997186711f9a199c2. Latest consensus state: current_term: 1 leader_uuid: "ab901ce6ae784c0997186711f9a199c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab901ce6ae784c0997186711f9a199c2" member_type: VOTER } }
I20261120 03:03:46.204461 4614 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:46.203714 4613 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab901ce6ae784c0997186711f9a199c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab901ce6ae784c0997186711f9a199c2" member_type: VOTER } }
I20261120 03:03:46.205269 4613 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:46.206921 4616 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:46.211404 4616 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:46.215711 385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:46.219229 4616 catalog_manager.cc:1398] Generated new cluster ID: 7f33f497200542aeb0de479e3371f4c8
I20261120 03:03:46.219445 4616 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:46.240929 4616 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:46.241993 4616 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:46.278982 4616 catalog_manager.cc:6237] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2: Generated new TSK 0
I20261120 03:03:46.279484 4616 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:46.346292 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:46.351475 4630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.352512 4631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.355122 385 server_base.cc:1061] running on GCE node
W20261120 03:03:46.355435 4633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.356325 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:46.356518 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:46.356652 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143826356641 us; error 0 us; skew 500 ppm
I20261120 03:03:46.357093 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:46.358989 385 webserver.cc:577] Webserver started at http://127.0.96.65:38183/ using document root <none> and password file <none>
I20261120 03:03:46.359369 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:46.359506 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:46.359691 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:46.360689 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "4013678e60394190a276ac0dbff7464b"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.364553 385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20261120 03:03:46.367406 4638 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.368072 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261120 03:03:46.368395 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "4013678e60394190a276ac0dbff7464b"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.368646 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:46.378721 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:46.379662 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:46.380973 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:46.382928 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:46.383136 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.383344 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:46.383500 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.418754 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:40039
I20261120 03:03:46.418843 4700 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:40039 every 8 connection(s)
I20261120 03:03:46.423411 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:46.429674 4705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.431241 4706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.432718 4708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.433269 385 server_base.cc:1061] running on GCE node
I20261120 03:03:46.434403 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:46.434674 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:46.434909 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143826434890 us; error 0 us; skew 500 ppm
I20261120 03:03:46.435056 4701 heartbeater.cc:345] Connected to a master server at 127.0.96.126:45111
I20261120 03:03:46.435346 4701 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:46.435487 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:46.436077 4701 heartbeater.cc:508] Master 127.0.96.126:45111 requested a full tablet report, sending...
I20261120 03:03:46.438010 385 webserver.cc:577] Webserver started at http://127.0.96.66:44291/ using document root <none> and password file <none>
I20261120 03:03:46.438421 4574 ts_manager.cc:210] Registered new tserver with Master: 4013678e60394190a276ac0dbff7464b (127.0.96.65:40039)
I20261120 03:03:46.438653 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:46.438913 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:46.439213 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:46.440573 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.440660 4574 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36936
I20261120 03:03:46.444908 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20261120 03:03:46.447921 4713 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.448626 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261120 03:03:46.448899 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.449146 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:46.468667 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:46.469552 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:46.470955 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:46.473026 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:46.473186 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.473457 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:46.473608 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.510665 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:34073
I20261120 03:03:46.510833 4775 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:34073 every 8 connection(s)
I20261120 03:03:46.515223 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:46.522864 4779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.523957 4780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.526901 4782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.527424 4776 heartbeater.cc:345] Connected to a master server at 127.0.96.126:45111
I20261120 03:03:46.527729 385 server_base.cc:1061] running on GCE node
I20261120 03:03:46.527851 4776 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:46.528656 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20261120 03:03:46.528703 4776 heartbeater.cc:508] Master 127.0.96.126:45111 requested a full tablet report, sending...
W20261120 03:03:46.528980 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:46.529259 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143826529241 us; error 0 us; skew 500 ppm
I20261120 03:03:46.529834 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:46.531131 4574 ts_manager.cc:210] Registered new tserver with Master: eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073)
I20261120 03:03:46.532344 385 webserver.cc:577] Webserver started at http://127.0.96.67:40297/ using document root <none> and password file <none>
I20261120 03:03:46.532967 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:46.532907 4574 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36942
I20261120 03:03:46.533360 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:46.533694 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:46.535100 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-2-root/instance:
uuid: "e68cec25598e47e3ab33cac75ae305da"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.539466 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20261120 03:03:46.542544 4787 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.543314 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20261120 03:03:46.543637 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-2-root
uuid: "e68cec25598e47e3ab33cac75ae305da"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.543902 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:46.559372 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:46.560307 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:46.561553 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:46.563509 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:46.563724 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.563954 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:46.564147 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.599982 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.67:44387
I20261120 03:03:46.600071 4849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.67:44387 every 8 connection(s)
I20261120 03:03:46.604602 385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:46.612017 4853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.613539 4854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.614902 385 server_base.cc:1061] running on GCE node
I20261120 03:03:46.614944 4850 heartbeater.cc:345] Connected to a master server at 127.0.96.126:45111
I20261120 03:03:46.615617 4850 heartbeater.cc:462] Registering TS with master...
W20261120 03:03:46.615926 4857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.616460 4850 heartbeater.cc:508] Master 127.0.96.126:45111 requested a full tablet report, sending...
I20261120 03:03:46.616858 385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:46.617084 385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:46.617261 385 hybrid_clock.cc:648] HybridClock initialized: now 1795143826617249 us; error 0 us; skew 500 ppm
I20261120 03:03:46.617911 385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:46.618525 4574 ts_manager.cc:210] Registered new tserver with Master: e68cec25598e47e3ab33cac75ae305da (127.0.96.67:44387)
I20261120 03:03:46.620026 4574 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36948
I20261120 03:03:46.620450 385 webserver.cc:577] Webserver started at http://127.0.96.68:41151/ using document root <none> and password file <none>
I20261120 03:03:46.621025 385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:46.621268 385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:46.621591 385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:46.622699 385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-3-root/instance:
uuid: "59b78942986249eda08d27d6863395d8"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.626894 385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20261120 03:03:46.629859 4861 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.630571 385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 03:03:46.630867 385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-3-root
uuid: "59b78942986249eda08d27d6863395d8"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.631152 385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:46.646580 385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 03:03:46.647521 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:46.648828 385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:46.650867 385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:46.651149 385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.651387 385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:46.651559 385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 03:03:46.687908 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.68:36041
I20261120 03:03:46.688004 4923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.68:36041 every 8 connection(s)
I20261120 03:03:46.699859 4924 heartbeater.cc:345] Connected to a master server at 127.0.96.126:45111
I20261120 03:03:46.700229 4924 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:46.700835 4924 heartbeater.cc:508] Master 127.0.96.126:45111 requested a full tablet report, sending...
I20261120 03:03:46.702544 4574 ts_manager.cc:210] Registered new tserver with Master: 59b78942986249eda08d27d6863395d8 (127.0.96.68:36041)
I20261120 03:03:46.702957 385 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.012188357s
I20261120 03:03:46.704056 4574 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36952
I20261120 03:03:46.739713 4574 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36976:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261120 03:03:46.777551 4889 tablet_service.cc:1495] Processing CreateTablet for tablet d195a20f388941cdb304ec869af88f73 (DEFAULT_TABLE table=test-workload [id=d13eb37246cc49babff26dd8a4d21252]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:46.778615 4889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d195a20f388941cdb304ec869af88f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:46.779963 4666 tablet_service.cc:1495] Processing CreateTablet for tablet d195a20f388941cdb304ec869af88f73 (DEFAULT_TABLE table=test-workload [id=d13eb37246cc49babff26dd8a4d21252]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:46.781162 4666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d195a20f388941cdb304ec869af88f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:46.781445 4741 tablet_service.cc:1495] Processing CreateTablet for tablet d195a20f388941cdb304ec869af88f73 (DEFAULT_TABLE table=test-workload [id=d13eb37246cc49babff26dd8a4d21252]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:46.782631 4741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d195a20f388941cdb304ec869af88f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:46.791404 4944 tablet_bootstrap.cc:492] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8: Bootstrap starting.
I20261120 03:03:46.799377 4944 tablet_bootstrap.cc:654] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:46.799803 4946 tablet_bootstrap.cc:492] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b: Bootstrap starting.
I20261120 03:03:46.801764 4945 tablet_bootstrap.cc:492] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: Bootstrap starting.
I20261120 03:03:46.804790 4944 tablet_bootstrap.cc:492] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8: No bootstrap required, opened a new log
I20261120 03:03:46.805271 4944 ts_tablet_manager.cc:1449] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20261120 03:03:46.807211 4946 tablet_bootstrap.cc:654] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:46.807979 4945 tablet_bootstrap.cc:654] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:46.808087 4944 raft_consensus.cc:366] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:46.808849 4944 raft_consensus.cc:392] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:46.809154 4944 raft_consensus.cc:787] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b78942986249eda08d27d6863395d8, State: Initialized, Role: FOLLOWER
I20261120 03:03:46.809904 4944 consensus_queue.cc:261] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:46.812773 4924 heartbeater.cc:500] Master 127.0.96.126:45111 was elected leader, sending a full tablet report...
I20261120 03:03:46.813289 4944 ts_tablet_manager.cc:1480] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20261120 03:03:46.813407 4946 tablet_bootstrap.cc:492] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b: No bootstrap required, opened a new log
I20261120 03:03:46.813866 4946 ts_tablet_manager.cc:1449] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20261120 03:03:46.814347 4945 tablet_bootstrap.cc:492] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: No bootstrap required, opened a new log
I20261120 03:03:46.814772 4945 ts_tablet_manager.cc:1449] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20261120 03:03:46.816360 4946 raft_consensus.cc:366] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:46.816911 4946 raft_consensus.cc:392] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:46.817147 4946 raft_consensus.cc:787] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4013678e60394190a276ac0dbff7464b, State: Initialized, Role: FOLLOWER
I20261120 03:03:46.817539 4945 raft_consensus.cc:366] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:46.817770 4946 consensus_queue.cc:261] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:46.818325 4945 raft_consensus.cc:392] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:46.818696 4945 raft_consensus.cc:787] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeee66a4d3b24e3a8c35ce5ab79e96ac, State: Initialized, Role: FOLLOWER
I20261120 03:03:46.819495 4945 consensus_queue.cc:261] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:46.822103 4776 heartbeater.cc:500] Master 127.0.96.126:45111 was elected leader, sending a full tablet report...
I20261120 03:03:46.822548 4945 ts_tablet_manager.cc:1480] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20261120 03:03:46.824457 4701 heartbeater.cc:500] Master 127.0.96.126:45111 was elected leader, sending a full tablet report...
I20261120 03:03:46.825717 4946 ts_tablet_manager.cc:1480] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20261120 03:03:47.009091 4952 raft_consensus.cc:534] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:47.009632 4952 raft_consensus.cc:556] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:47.012039 4952 leader_election.cc:290] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59b78942986249eda08d27d6863395d8 (127.0.96.68:36041), 4013678e60394190a276ac0dbff7464b (127.0.96.65:40039)
I20261120 03:03:47.020725 4899 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b78942986249eda08d27d6863395d8" is_pre_election: true
I20261120 03:03:47.021443 4899 raft_consensus.cc:2062] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate eeee66a4d3b24e3a8c35ce5ab79e96ac in term 0
I20261120 03:03:47.021900 4676 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4013678e60394190a276ac0dbff7464b" is_pre_election: true
I20261120 03:03:47.022379 4717 leader_election.cc:304] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b78942986249eda08d27d6863395d8, eeee66a4d3b24e3a8c35ce5ab79e96ac; no voters:
I20261120 03:03:47.022612 4676 raft_consensus.cc:2062] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate eeee66a4d3b24e3a8c35ce5ab79e96ac in term 0
I20261120 03:03:47.023064 4952 raft_consensus.cc:3024] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:47.023484 4952 raft_consensus.cc:534] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:47.023818 4952 raft_consensus.cc:3296] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:47.024899 4952 raft_consensus.cc:556] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:47.030076 4952 leader_election.cc:290] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [CANDIDATE]: Term 1 election: Requested vote from peers 59b78942986249eda08d27d6863395d8 (127.0.96.68:36041), 4013678e60394190a276ac0dbff7464b (127.0.96.65:40039)
I20261120 03:03:47.030712 4899 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b78942986249eda08d27d6863395d8"
I20261120 03:03:47.031041 4676 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4013678e60394190a276ac0dbff7464b"
I20261120 03:03:47.031157 4899 raft_consensus.cc:3296] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:47.031530 4676 raft_consensus.cc:3296] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:47.032027 4899 raft_consensus.cc:2062] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate eeee66a4d3b24e3a8c35ce5ab79e96ac in term 1
I20261120 03:03:47.032428 4676 raft_consensus.cc:2062] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate eeee66a4d3b24e3a8c35ce5ab79e96ac in term 1
I20261120 03:03:47.036778 4717 leader_election.cc:304] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b78942986249eda08d27d6863395d8, eeee66a4d3b24e3a8c35ce5ab79e96ac; no voters:
I20261120 03:03:47.037560 4952 raft_consensus.cc:3024] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:47.037981 4952 raft_consensus.cc:744] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 1 LEADER]: Becoming Leader. State: Replica: eeee66a4d3b24e3a8c35ce5ab79e96ac, State: Running, Role: LEADER
I20261120 03:03:47.038738 4952 consensus_queue.cc:238] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:47.044446 4952 consensus_queue.cc:1049] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:47.045560 4956 consensus_queue.cc:1049] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:47.045956 4573 catalog_manager.cc:5856] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac reported cstate change: term changed from 0 to 1, leader changed from <none> to eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66). New cstate: current_term: 1 leader_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:47.296648 4666 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:47.301026 4889 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:47.309549 4741 tablet_service.cc:1451] Tablet server has 1 leaders and 0 scanners
I20261120 03:03:47.311543 4815 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
ab901ce6ae784c0997186711f9a199c2 | 127.0.96.126:45111 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
4013678e60394190a276ac0dbff7464b | 127.0.96.65:40039 | HEALTHY | <none> | 0 | 0
59b78942986249eda08d27d6863395d8 | 127.0.96.68:36041 | HEALTHY | <none> | 0 | 0
e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | HEALTHY | <none> | 0 | 0
eeee66a4d3b24e3a8c35ce5ab79e96ac | 127.0.96.66:34073 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20261120 03:03:47.579432 385 log_verifier.cc:126] Checking tablet d195a20f388941cdb304ec869af88f73
I20261120 03:03:47.622825 4850 heartbeater.cc:500] Master 127.0.96.126:45111 was elected leader, sending a full tablet report...
I20261120 03:03:47.636984 385 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d195a20f388941cdb304ec869af88f73
I20261120 03:03:47.638901 385 tablet_replica.cc:333] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: stopping tablet replica
I20261120 03:03:47.639436 385 raft_consensus.cc:2440] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:47.640311 385 raft_consensus.cc:2469] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:47.860884 4889 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:47.863045 4666 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:47.863466 4741 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:47.868448 4815 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
ab901ce6ae784c0997186711f9a199c2 | 127.0.96.126:45111 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
4013678e60394190a276ac0dbff7464b | 127.0.96.65:40039 | HEALTHY | <none> | 0 | 0
59b78942986249eda08d27d6863395d8 | 127.0.96.68:36041 | HEALTHY | <none> | 0 | 0
e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | HEALTHY | <none> | 0 | 0
eeee66a4d3b24e3a8c35ce5ab79e96ac | 127.0.96.66:34073 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet d195a20f388941cdb304ec869af88f73 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
59b78942986249eda08d27d6863395d8 (127.0.96.68:36041): RUNNING
4013678e60394190a276ac0dbff7464b (127.0.96.65:40039): RUNNING
eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = 59b78942986249eda08d27d6863395d8
B = 4013678e60394190a276ac0dbff7464b
C = eeee66a4d3b24e3a8c35ce5ab79e96ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20261120 03:03:48.099592 385 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261120 03:03:48.459100 4666 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:48.465679 4889 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:48.467797 4815 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:48.469378 4741 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:48.568826 5050 raft_consensus.cc:534] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeee66a4d3b24e3a8c35ce5ab79e96ac)
I20261120 03:03:48.578768 5050 raft_consensus.cc:556] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:48.600183 5050 leader_election.cc:290] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4013678e60394190a276ac0dbff7464b (127.0.96.65:40039), eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073)
I20261120 03:03:48.629886 4751 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "59b78942986249eda08d27d6863395d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" is_pre_election: true
W20261120 03:03:48.631368 4863 leader_election.cc:343] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): Illegal state: cannot vote while shut down
I20261120 03:03:48.631697 4676 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "59b78942986249eda08d27d6863395d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4013678e60394190a276ac0dbff7464b" is_pre_election: true
I20261120 03:03:48.633631 4864 leader_election.cc:304] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59b78942986249eda08d27d6863395d8; no voters: 4013678e60394190a276ac0dbff7464b, eeee66a4d3b24e3a8c35ce5ab79e96ac
I20261120 03:03:48.634711 5050 raft_consensus.cc:2966] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
ab901ce6ae784c0997186711f9a199c2 | 127.0.96.126:45111 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
4013678e60394190a276ac0dbff7464b | 127.0.96.65:40039 | HEALTHY | <none> | 0 | 0
59b78942986249eda08d27d6863395d8 | 127.0.96.68:36041 | HEALTHY | <none> | 0 | 0
e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | HEALTHY | <none> | 0 | 0
eeee66a4d3b24e3a8c35ce5ab79e96ac | 127.0.96.66:34073 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet d195a20f388941cdb304ec869af88f73 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
59b78942986249eda08d27d6863395d8 (127.0.96.68:36041): RUNNING
4013678e60394190a276ac0dbff7464b (127.0.96.65:40039): RUNNING
eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = 59b78942986249eda08d27d6863395d8
B = 4013678e60394190a276ac0dbff7464b
C = eeee66a4d3b24e3a8c35ce5ab79e96ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20261120 03:03:48.717234 385 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261120 03:03:49.031080 5060 raft_consensus.cc:534] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeee66a4d3b24e3a8c35ce5ab79e96ac)
I20261120 03:03:49.031570 5060 raft_consensus.cc:556] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:49.033778 5060 leader_election.cc:290] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59b78942986249eda08d27d6863395d8 (127.0.96.68:36041), eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073)
I20261120 03:03:49.044205 4899 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "4013678e60394190a276ac0dbff7464b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59b78942986249eda08d27d6863395d8" is_pre_election: true
I20261120 03:03:49.044996 4899 raft_consensus.cc:2062] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 4013678e60394190a276ac0dbff7464b in term 1
I20261120 03:03:49.046286 4642 leader_election.cc:304] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4013678e60394190a276ac0dbff7464b, 59b78942986249eda08d27d6863395d8; no voters:
I20261120 03:03:49.047107 5060 raft_consensus.cc:3024] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:49.047014 4751 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "4013678e60394190a276ac0dbff7464b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" is_pre_election: true
I20261120 03:03:49.048115 5060 raft_consensus.cc:534] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeee66a4d3b24e3a8c35ce5ab79e96ac)
W20261120 03:03:49.048427 4640 leader_election.cc:343] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): Illegal state: cannot vote while shut down
I20261120 03:03:49.048599 5060 raft_consensus.cc:3296] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:49.050215 5060 raft_consensus.cc:556] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:49.057989 5060 leader_election.cc:290] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [CANDIDATE]: Term 2 election: Requested vote from peers 59b78942986249eda08d27d6863395d8 (127.0.96.68:36041), eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073)
I20261120 03:03:49.058753 4899 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "4013678e60394190a276ac0dbff7464b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59b78942986249eda08d27d6863395d8"
I20261120 03:03:49.059052 4751 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "4013678e60394190a276ac0dbff7464b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac"
I20261120 03:03:49.059221 4899 raft_consensus.cc:3296] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 1 FOLLOWER]: Advancing to term 2
W20261120 03:03:49.060195 4640 leader_election.cc:343] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): Illegal state: cannot vote while shut down
I20261120 03:03:49.063990 4899 raft_consensus.cc:2062] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate 4013678e60394190a276ac0dbff7464b in term 2
I20261120 03:03:49.064875 4642 leader_election.cc:304] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4013678e60394190a276ac0dbff7464b, 59b78942986249eda08d27d6863395d8; no voters: eeee66a4d3b24e3a8c35ce5ab79e96ac
I20261120 03:03:49.065515 5060 raft_consensus.cc:3024] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:49.065984 5060 raft_consensus.cc:744] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 LEADER]: Becoming Leader. State: Replica: 4013678e60394190a276ac0dbff7464b, State: Running, Role: LEADER
I20261120 03:03:49.066710 5060 consensus_queue.cc:238] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:49.073774 4574 catalog_manager.cc:5856] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b reported cstate change: term changed from 1 to 2, leader changed from eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66) to 4013678e60394190a276ac0dbff7464b (127.0.96.65). New cstate: current_term: 2 leader_uuid: "4013678e60394190a276ac0dbff7464b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:49.077309 4899 raft_consensus.cc:1338] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Refusing update from remote peer 4013678e60394190a276ac0dbff7464b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261120 03:03:49.078903 5064 consensus_queue.cc:1049] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 03:03:49.080024 4640 consensus_queue.cc:580] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: The tablet replica hosted on peer eeee66a4d3b24e3a8c35ce5ab79e96ac has failed
W20261120 03:03:49.080476 4640 consensus_peers.cc:597] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b -> Peer eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): Couldn't send request to peer eeee66a4d3b24e3a8c35ce5ab79e96ac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:49.118326 4676 consensus_queue.cc:238] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } }
I20261120 03:03:49.122988 4899 raft_consensus.cc:1338] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Refusing update from remote peer 4013678e60394190a276ac0dbff7464b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:49.124249 5060 consensus_queue.cc:1049] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:49.129558 5066 raft_consensus.cc:3183] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } }
I20261120 03:03:49.132598 4560 catalog_manager.cc:5263] ChangeConfig:REMOVE_PEER RPC for tablet d195a20f388941cdb304ec869af88f73 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20261120 03:03:49.136091 4573 catalog_manager.cc:5856] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b reported cstate change: config changed from index -1 to 3, VOTER eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66) evicted. New cstate: current_term: 2 leader_uuid: "4013678e60394190a276ac0dbff7464b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:49.138659 4899 raft_consensus.cc:3183] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } }
I20261120 03:03:49.152062 4741 tablet_service.cc:1542] Processing DeleteTablet for tablet d195a20f388941cdb304ec869af88f73 with delete_type TABLET_DATA_TOMBSTONED (TS eeee66a4d3b24e3a8c35ce5ab79e96ac not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43146
I20261120 03:03:49.153717 5086 ts_tablet_manager.cc:1962] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20261120 03:03:49.156804 4676 consensus_queue.cc:238] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "e68cec25598e47e3ab33cac75ae305da" member_type: NON_VOTER last_known_addr { host: "127.0.96.67" port: 44387 } attrs { promote: true } }
I20261120 03:03:49.165867 4899 raft_consensus.cc:1338] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Refusing update from remote peer 4013678e60394190a276ac0dbff7464b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20261120 03:03:49.166860 5086 ts_tablet_manager.cc:1975] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20261120 03:03:49.167259 5086 log.cc:1199] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: Deleting WAL directory at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-1-root/wals/d195a20f388941cdb304ec869af88f73
I20261120 03:03:49.167615 5064 consensus_queue.cc:1049] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20261120 03:03:49.169291 4559 catalog_manager.cc:5081] TS eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): tablet d195a20f388941cdb304ec869af88f73 (table test-workload [id=d13eb37246cc49babff26dd8a4d21252]) successfully deleted
I20261120 03:03:49.173926 5066 raft_consensus.cc:3183] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER e68cec25598e47e3ab33cac75ae305da (127.0.96.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "e68cec25598e47e3ab33cac75ae305da" member_type: NON_VOTER last_known_addr { host: "127.0.96.67" port: 44387 } attrs { promote: true } } }
I20261120 03:03:49.176324 4560 catalog_manager.cc:5263] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d195a20f388941cdb304ec869af88f73 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20261120 03:03:49.175645 4899 raft_consensus.cc:3183] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER e68cec25598e47e3ab33cac75ae305da (127.0.96.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "e68cec25598e47e3ab33cac75ae305da" member_type: NON_VOTER last_known_addr { host: "127.0.96.67" port: 44387 } attrs { promote: true } } }
W20261120 03:03:49.180035 4641 consensus_peers.cc:597] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b -> Peer e68cec25598e47e3ab33cac75ae305da (127.0.96.67:44387): Couldn't send request to peer e68cec25598e47e3ab33cac75ae305da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d195a20f388941cdb304ec869af88f73. This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:49.181865 4574 catalog_manager.cc:5856] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b reported cstate change: config changed from index 3 to 4, NON_VOTER e68cec25598e47e3ab33cac75ae305da (127.0.96.67) added. New cstate: current_term: 2 leader_uuid: "4013678e60394190a276ac0dbff7464b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e68cec25598e47e3ab33cac75ae305da" member_type: NON_VOTER last_known_addr { host: "127.0.96.67" port: 44387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:49.224613 4666 tablet_service.cc:1451] Tablet server has 1 leaders and 0 scanners
I20261120 03:03:49.229179 4889 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:49.236581 4815 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:49.238199 4741 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
ab901ce6ae784c0997186711f9a199c2 | 127.0.96.126:45111 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
4013678e60394190a276ac0dbff7464b | 127.0.96.65:40039 | HEALTHY | <none> | 1 | 0
59b78942986249eda08d27d6863395d8 | 127.0.96.68:36041 | HEALTHY | <none> | 0 | 0
e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | HEALTHY | <none> | 0 | 0
eeee66a4d3b24e3a8c35ce5ab79e96ac | 127.0.96.66:34073 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | eeee66a4d3b24e3a8c35ce5ab79e96ac | 127.0.96.66:34073 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20261120 03:03:49.551662 385 log_verifier.cc:126] Checking tablet d195a20f388941cdb304ec869af88f73
I20261120 03:03:49.587256 385 log_verifier.cc:177] Verified matching terms for 4 ops in tablet d195a20f388941cdb304ec869af88f73
I20261120 03:03:49.591552 385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:49.615511 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:49.616271 385 tablet_replica.cc:333] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b: stopping tablet replica
I20261120 03:03:49.616806 385 raft_consensus.cc:2440] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 LEADER]: Raft consensus shutting down.
I20261120 03:03:49.617897 385 raft_consensus.cc:2469] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:49.630429 385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:49.644192 385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:49.663689 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:49.681990 385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:49.693478 385 tablet_server.cc:179] TabletServer@127.0.96.67:0 shutting down...
I20261120 03:03:49.714630 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:49.730579 385 tablet_server.cc:196] TabletServer@127.0.96.67:0 shutdown complete.
I20261120 03:03:49.739852 385 tablet_server.cc:179] TabletServer@127.0.96.68:0 shutting down...
I20261120 03:03:49.761176 385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:49.761825 385 tablet_replica.cc:333] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8: stopping tablet replica
I20261120 03:03:49.762534 385 raft_consensus.cc:2440] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:49.763254 385 raft_consensus.cc:2469] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:49.783582 385 tablet_server.cc:196] TabletServer@127.0.96.68:0 shutdown complete.
I20261120 03:03:49.794668 385 master.cc:585] Master@127.0.96.126:45111 shutting down...
I20261120 03:03:49.814589 385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:49.815093 385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:49.815389 385 tablet_replica.cc:333] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2: stopping tablet replica
I20261120 03:03:49.833451 385 master.cc:607] Master@127.0.96.126:45111 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (3764 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (8566 ms total)
[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (60981 ms total)
[ PASSED ] 11 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[ FAILED ] 1 test, listed below:
[ FAILED ] TsTabletManagerITest.TestTableStats
1 FAILED TEST
I20261120 03:03:49.872270 385 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~9 seconds ago
I20261120 03:03:49.872552 385 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 6 messages since previous log ~13 seconds ago
I20261120 03:03:49.872710 385 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:648: suppressed but not reported on 72 messages since previous log ~56 seconds ago
I20261120 03:03:49.872943 385 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2554: suppressed but not reported on 7 messages since previous log ~12 seconds ago