[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20250905 19:59:05.777942 15038 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.158s user 0.442s sys 0.713s
W20250905 19:59:05.778455 15038 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.159s user 0.442s sys 0.713s
I20250905 19:59:05.791842 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:38789
I20250905 19:59:05.793498 15038 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 19:59:05.794787 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:05.806967 15049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:05.809691 15050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:05.899027 15052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:05.899358 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:05.900497 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:05.900702 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:05.900878 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102345900857 us; error 0 us; skew 500 ppm
I20250905 19:59:05.901655 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:05.906140 15038 webserver.cc:480] Webserver started at http://127.14.175.190:33831/ using document root <none> and password file <none>
I20250905 19:59:05.906966 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:05.907162 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:05.907624 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:05.912904 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "b9186063ede54c8cbdf48ec3d60d4b67"
format_stamp: "Formatted at 2025-09-05 19:59:05 on dist-test-slave-22s0"
I20250905 19:59:05.920006 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250905 19:59:05.924805 15057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:05.925822 15038 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250905 19:59:05.926133 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "b9186063ede54c8cbdf48ec3d60d4b67"
format_stamp: "Formatted at 2025-09-05 19:59:05 on dist-test-slave-22s0"
I20250905 19:59:05.926452 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:05.942098 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:05.943490 15038 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 19:59:05.943902 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:06.013043 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:38789
I20250905 19:59:06.013110 15108 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:38789 every 8 connection(s)
I20250905 19:59:06.019060 15109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:06.040597 15109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67: Bootstrap starting.
I20250905 19:59:06.046669 15109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:06.048322 15109 log.cc:826] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67: Log is configured to *not* fsync() on all Append() calls
I20250905 19:59:06.052428 15109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67: No bootstrap required, opened a new log
I20250905 19:59:06.072413 15109 raft_consensus.cc:357] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9186063ede54c8cbdf48ec3d60d4b67" member_type: VOTER }
I20250905 19:59:06.072957 15109 raft_consensus.cc:383] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:06.073170 15109 raft_consensus.cc:738] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9186063ede54c8cbdf48ec3d60d4b67, State: Initialized, Role: FOLLOWER
I20250905 19:59:06.073946 15109 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9186063ede54c8cbdf48ec3d60d4b67" member_type: VOTER }
I20250905 19:59:06.074401 15109 raft_consensus.cc:397] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:06.074680 15109 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:06.074959 15109 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:06.078773 15109 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9186063ede54c8cbdf48ec3d60d4b67" member_type: VOTER }
I20250905 19:59:06.079387 15109 leader_election.cc:304] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9186063ede54c8cbdf48ec3d60d4b67; no voters:
I20250905 19:59:06.080883 15109 leader_election.cc:290] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:06.081264 15112 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:06.083340 15112 raft_consensus.cc:695] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [term 1 LEADER]: Becoming Leader. State: Replica: b9186063ede54c8cbdf48ec3d60d4b67, State: Running, Role: LEADER
I20250905 19:59:06.084144 15112 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9186063ede54c8cbdf48ec3d60d4b67" member_type: VOTER }
I20250905 19:59:06.085198 15109 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:06.090996 15114 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9186063ede54c8cbdf48ec3d60d4b67. Latest consensus state: current_term: 1 leader_uuid: "b9186063ede54c8cbdf48ec3d60d4b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9186063ede54c8cbdf48ec3d60d4b67" member_type: VOTER } }
I20250905 19:59:06.091122 15113 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9186063ede54c8cbdf48ec3d60d4b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9186063ede54c8cbdf48ec3d60d4b67" member_type: VOTER } }
I20250905 19:59:06.091701 15114 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:06.091764 15113 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:06.098030 15119 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:06.110080 15119 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:06.111928 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:59:06.125473 15119 catalog_manager.cc:1349] Generated new cluster ID: 5860556c8ba643a0b8a27ae4796dea79
I20250905 19:59:06.125717 15119 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:06.143589 15119 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:59:06.145218 15119 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:06.160086 15119 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67: Generated new TSK 0
I20250905 19:59:06.160955 15119 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:06.178212 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:06.186188 15131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:06.187484 15132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:06.191032 15134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:06.191284 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:06.192076 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:06.192263 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:06.192425 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102346192405 us; error 0 us; skew 500 ppm
I20250905 19:59:06.192960 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:06.196719 15038 webserver.cc:480] Webserver started at http://127.14.175.129:43303/ using document root <none> and password file <none>
I20250905 19:59:06.197225 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:06.197400 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:06.197669 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:06.198916 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "ac1ba6635696411790dd9f88b814fcda"
format_stamp: "Formatted at 2025-09-05 19:59:06 on dist-test-slave-22s0"
I20250905 19:59:06.204553 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250905 19:59:06.208997 15139 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:06.209834 15038 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 19:59:06.210090 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "ac1ba6635696411790dd9f88b814fcda"
format_stamp: "Formatted at 2025-09-05 19:59:06 on dist-test-slave-22s0"
I20250905 19:59:06.210371 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:06.226001 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:06.227355 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:06.230131 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:06.234624 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:06.234820 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:06.235088 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:06.235253 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:06.338936 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:42049
I20250905 19:59:06.338994 15201 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:42049 every 8 connection(s)
I20250905 19:59:06.345986 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:06.357862 15206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:06.361805 15207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:06.372188 15209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:06.377640 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:06.378933 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:06.379201 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:06.379400 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102346379384 us; error 0 us; skew 500 ppm
I20250905 19:59:06.380067 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:06.383631 15038 webserver.cc:480] Webserver started at http://127.14.175.130:36301/ using document root <none> and password file <none>
I20250905 19:59:06.384258 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:06.384488 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:06.384794 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:06.386363 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/ts-1-root/instance:
uuid: "6ad4db79bc5e462a988e8e57f6af5b6c"
format_stamp: "Formatted at 2025-09-05 19:59:06 on dist-test-slave-22s0"
I20250905 19:59:06.393052 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250905 19:59:06.398386 15202 heartbeater.cc:344] Connected to a master server at 127.14.175.190:38789
I20250905 19:59:06.398628 15214 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:06.398988 15202 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:06.399699 15038 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250905 19:59:06.400033 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/ts-1-root
uuid: "6ad4db79bc5e462a988e8e57f6af5b6c"
format_stamp: "Formatted at 2025-09-05 19:59:06 on dist-test-slave-22s0"
I20250905 19:59:06.400337 15202 heartbeater.cc:507] Master 127.14.175.190:38789 requested a full tablet report, sending...
I20250905 19:59:06.400408 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1757102344413625-15038-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:06.404071 15074 ts_manager.cc:194] Registered new tserver with Master: ac1ba6635696411790dd9f88b814fcda (127.14.175.129:42049)
I20250905 19:59:06.407507 15074 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41534
I20250905 19:59:06.438140 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:06.439893 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:06.441664 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:06.445567 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:06.445822 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:06.446102 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:06.446342 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:06.511817 15202 heartbeater.cc:499] Master 127.14.175.190:38789 was elected leader, sending a full tablet report...
I20250905 19:59:06.537432 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.130:33675
I20250905 19:59:06.537606 15276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.130:33675 every 8 connection(s)
I20250905 19:59:06.554375 15277 heartbeater.cc:344] Connected to a master server at 127.14.175.190:38789
I20250905 19:59:06.554796 15277 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:06.555729 15277 heartbeater.cc:507] Master 127.14.175.190:38789 requested a full tablet report, sending...
I20250905 19:59:06.557557 15074 ts_manager.cc:194] Registered new tserver with Master: 6ad4db79bc5e462a988e8e57f6af5b6c (127.14.175.130:33675)
I20250905 19:59:06.558597 15038 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.014925475s
I20250905 19:59:06.558677 15074 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55772
I20250905 19:59:06.587829 15074 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:59:06.605324 15074 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20250905 19:59:06.659938 15167 tablet_service.cc:1468] Processing CreateTablet for tablet b5faca015ad4488c8c2facc5f97a3ac7 (DEFAULT_TABLE table=test-table [id=9e63c58fac97471c8ccff5cda7c25c2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:06.660729 15277 heartbeater.cc:499] Master 127.14.175.190:38789 was elected leader, sending a full tablet report...
I20250905 19:59:06.659927 15166 tablet_service.cc:1468] Processing CreateTablet for tablet 7e5ace6046144c9e91240338c6cfb93b (DEFAULT_TABLE table=test-table [id=9e63c58fac97471c8ccff5cda7c25c2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:06.661715 15167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5faca015ad4488c8c2facc5f97a3ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:06.662487 15166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e5ace6046144c9e91240338c6cfb93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:06.665589 15242 tablet_service.cc:1468] Processing CreateTablet for tablet b5faca015ad4488c8c2facc5f97a3ac7 (DEFAULT_TABLE table=test-table [id=9e63c58fac97471c8ccff5cda7c25c2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:06.666833 15242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5faca015ad4488c8c2facc5f97a3ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:06.667493 15241 tablet_service.cc:1468] Processing CreateTablet for tablet 7e5ace6046144c9e91240338c6cfb93b (DEFAULT_TABLE table=test-table [id=9e63c58fac97471c8ccff5cda7c25c2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:06.668699 15241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e5ace6046144c9e91240338c6cfb93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:06.684947 15290 tablet_bootstrap.cc:492] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda: Bootstrap starting.
I20250905 19:59:06.689777 15291 tablet_bootstrap.cc:492] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c: Bootstrap starting.
I20250905 19:59:06.694362 15290 tablet_bootstrap.cc:654] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:06.698076 15291 tablet_bootstrap.cc:654] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:06.704188 15290 tablet_bootstrap.cc:492] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda: No bootstrap required, opened a new log
I20250905 19:59:06.704559 15290 ts_tablet_manager.cc:1397] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.000s
I20250905 19:59:06.706466 15290 raft_consensus.cc:357] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:06.707260 15290 raft_consensus.cc:738] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1ba6635696411790dd9f88b814fcda, State: Initialized, Role: FOLLOWER
I20250905 19:59:06.707880 15290 consensus_queue.cc:260] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:06.709112 15291 tablet_bootstrap.cc:492] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c: No bootstrap required, opened a new log
I20250905 19:59:06.709553 15291 ts_tablet_manager.cc:1397] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.003s
I20250905 19:59:06.710382 15290 ts_tablet_manager.cc:1428] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250905 19:59:06.711966 15290 tablet_bootstrap.cc:492] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda: Bootstrap starting.
I20250905 19:59:06.714290 15291 raft_consensus.cc:357] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:06.714977 15291 raft_consensus.cc:738] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad4db79bc5e462a988e8e57f6af5b6c, State: Initialized, Role: FOLLOWER
I20250905 19:59:06.715581 15291 consensus_queue.cc:260] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:06.719070 15290 tablet_bootstrap.cc:654] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:06.728780 15291 ts_tablet_manager.cc:1428] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
I20250905 19:59:06.730881 15291 tablet_bootstrap.cc:492] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c: Bootstrap starting.
I20250905 19:59:06.732286 15290 tablet_bootstrap.cc:492] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda: No bootstrap required, opened a new log
I20250905 19:59:06.732712 15290 ts_tablet_manager.cc:1397] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.005s
I20250905 19:59:06.735384 15290 raft_consensus.cc:357] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:06.735996 15290 raft_consensus.cc:738] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1ba6635696411790dd9f88b814fcda, State: Initialized, Role: FOLLOWER
I20250905 19:59:06.736542 15290 consensus_queue.cc:260] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:06.737192 15291 tablet_bootstrap.cc:654] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:06.738974 15290 ts_tablet_manager.cc:1428] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250905 19:59:06.746384 15291 tablet_bootstrap.cc:492] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c: No bootstrap required, opened a new log
I20250905 19:59:06.746845 15291 ts_tablet_manager.cc:1397] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20250905 19:59:06.749451 15291 raft_consensus.cc:357] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:06.750075 15291 raft_consensus.cc:738] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad4db79bc5e462a988e8e57f6af5b6c, State: Initialized, Role: FOLLOWER
I20250905 19:59:06.750671 15291 consensus_queue.cc:260] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:06.752826 15291 ts_tablet_manager.cc:1428] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20250905 19:59:06.775698 15038 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20250905 19:59:06.776015 15038 raft_consensus.cc:421] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [term 0 FOLLOWER]: Emulating election...
I20250905 19:59:06.776260 15038 raft_consensus.cc:3058] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:06.779949 15038 raft_consensus.cc:695] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [term 1 LEADER]: Becoming Leader. State: Replica: 6ad4db79bc5e462a988e8e57f6af5b6c, State: Running, Role: LEADER
I20250905 19:59:06.780591 15038 consensus_queue.cc:237] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:06.783892 15038 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250905 19:59:06.790112 15074 catalog_manager.cc:5582] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ad4db79bc5e462a988e8e57f6af5b6c (127.14.175.130). New cstate: current_term: 1 leader_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } health_report { overall_health: HEALTHY } } }
W20250905 19:59:06.795608 15278 tablet.cc:2378] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 19:59:06.808027 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20250905 19:59:06.911801 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20250905 19:59:07.115233 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20250905 19:59:07.336548 15177 raft_consensus.cc:3058] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:07.343546 15301 consensus_queue.cc:1035] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:59:07.418905 15038 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20250905 19:59:07.419152 15038 raft_consensus.cc:421] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 0 FOLLOWER]: Emulating election...
I20250905 19:59:07.419373 15038 raft_consensus.cc:3058] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:07.423619 15038 raft_consensus.cc:695] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 1 LEADER]: Becoming Leader. State: Replica: 6ad4db79bc5e462a988e8e57f6af5b6c, State: Running, Role: LEADER
I20250905 19:59:07.424229 15038 consensus_queue.cc:237] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:07.426422 15038 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250905 19:59:07.430279 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20250905 19:59:07.430106 15073 catalog_manager.cc:5582] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ad4db79bc5e462a988e8e57f6af5b6c (127.14.175.130). New cstate: current_term: 1 leader_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:07.533906 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20250905 19:59:07.737475 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20250905 19:59:07.889453 15177 raft_consensus.cc:3058] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:07.894714 15309 consensus_queue.cc:1035] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:59:08.242646 15038 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20250905 19:59:08.242883 15038 raft_consensus.cc:421] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 1 LEADER]: Emulating election...
I20250905 19:59:08.243117 15038 raft_consensus.cc:3053] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:08.243297 15038 raft_consensus.cc:738] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ad4db79bc5e462a988e8e57f6af5b6c, State: Running, Role: LEADER
I20250905 19:59:08.243750 15038 consensus_queue.cc:260] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:08.244455 15038 raft_consensus.cc:3058] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:08.248181 15038 raft_consensus.cc:695] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 2 LEADER]: Becoming Leader. State: Replica: 6ad4db79bc5e462a988e8e57f6af5b6c, State: Running, Role: LEADER
I20250905 19:59:08.248744 15038 consensus_queue.cc:237] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:08.250092 15038 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250905 19:59:08.253590 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 1.1 2.2
I20250905 19:59:08.253744 15072 catalog_manager.cc:5582] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:08.356683 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 1.1 2.2
I20250905 19:59:08.560521 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 1.1 2.2
I20250905 19:59:08.685638 15177 raft_consensus.cc:3058] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:08.690284 15177 raft_consensus.cc:1273] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 2 FOLLOWER]: Refusing update from remote peer 6ad4db79bc5e462a988e8e57f6af5b6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:08.691490 15309 consensus_queue.cc:1035] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:08.864053 15038 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20250905 19:59:08.864308 15038 raft_consensus.cc:421] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 2 FOLLOWER]: Emulating election...
I20250905 19:59:08.864529 15038 raft_consensus.cc:3058] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:08.868706 15038 raft_consensus.cc:695] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 3 LEADER]: Becoming Leader. State: Replica: ac1ba6635696411790dd9f88b814fcda, State: Running, Role: LEADER
I20250905 19:59:08.869329 15038 consensus_queue.cc:237] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:08.870932 15038 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250905 19:59:08.874647 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 2.2
I20250905 19:59:08.874161 15073 catalog_manager.cc:5582] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda reported cstate change: term changed from 2 to 3, leader changed from 6ad4db79bc5e462a988e8e57f6af5b6c (127.14.175.130) to ac1ba6635696411790dd9f88b814fcda (127.14.175.129). New cstate: current_term: 3 leader_uuid: "ac1ba6635696411790dd9f88b814fcda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:08.977993 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 2.2
I20250905 19:59:09.181857 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 2.2
I20250905 19:59:09.208905 15177 raft_consensus.cc:1238] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 3 LEADER]: Rejecting Update request from peer 6ad4db79bc5e462a988e8e57f6af5b6c for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:09.210063 15323 consensus_queue.cc:1046] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:09.211081 15323 consensus_queue.cc:1230] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [LEADER]: Peer ac1ba6635696411790dd9f88b814fcda log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250905 19:59:09.211467 15329 raft_consensus.cc:3053] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:09.211730 15329 raft_consensus.cc:738] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ad4db79bc5e462a988e8e57f6af5b6c, State: Running, Role: LEADER
I20250905 19:59:09.212371 15329 consensus_queue.cc:260] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:09.213083 15329 raft_consensus.cc:3058] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:09.481108 15252 raft_consensus.cc:1273] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 3 FOLLOWER]: Refusing update from remote peer ac1ba6635696411790dd9f88b814fcda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:09.482369 15330 consensus_queue.cc:1035] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:09.686050 15038 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20250905 19:59:09.686347 15038 raft_consensus.cc:421] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda [term 1 FOLLOWER]: Emulating election...
I20250905 19:59:09.686650 15038 raft_consensus.cc:3058] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:09.690871 15038 raft_consensus.cc:695] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda [term 2 LEADER]: Becoming Leader. State: Replica: ac1ba6635696411790dd9f88b814fcda, State: Running, Role: LEADER
I20250905 19:59:09.691524 15038 consensus_queue.cc:237] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:09.693415 15038 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250905 19:59:09.697018 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 1.1
I20250905 19:59:09.696856 15073 catalog_manager.cc:5582] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda reported cstate change: term changed from 1 to 2, leader changed from 6ad4db79bc5e462a988e8e57f6af5b6c (127.14.175.130) to ac1ba6635696411790dd9f88b814fcda (127.14.175.129). New cstate: current_term: 2 leader_uuid: "ac1ba6635696411790dd9f88b814fcda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:09.800848 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 1.1
I20250905 19:59:09.977632 15177 raft_consensus.cc:1238] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda [term 2 LEADER]: Rejecting Update request from peer 6ad4db79bc5e462a988e8e57f6af5b6c for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:09.978835 15329 consensus_queue.cc:1046] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:09.980361 15329 consensus_queue.cc:1230] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [LEADER]: Peer ac1ba6635696411790dd9f88b814fcda log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250905 19:59:09.981062 15340 raft_consensus.cc:3053] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:09.981551 15340 raft_consensus.cc:738] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ad4db79bc5e462a988e8e57f6af5b6c, State: Running, Role: LEADER
I20250905 19:59:09.982125 15340 consensus_queue.cc:260] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:09.982792 15340 raft_consensus.cc:3058] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:10.004690 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 1.1
I20250905 19:59:10.208436 15252 raft_consensus.cc:1273] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [term 2 FOLLOWER]: Refusing update from remote peer ac1ba6635696411790dd9f88b814fcda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:10.209451 15333 consensus_queue.cc:1035] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:10.308287 15038 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20250905 19:59:10.308526 15038 raft_consensus.cc:421] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 3 FOLLOWER]: Emulating election...
I20250905 19:59:10.308724 15038 raft_consensus.cc:3058] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:10.312724 15038 raft_consensus.cc:695] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 4 LEADER]: Becoming Leader. State: Replica: 6ad4db79bc5e462a988e8e57f6af5b6c, State: Running, Role: LEADER
I20250905 19:59:10.313236 15038 consensus_queue.cc:237] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:10.314550 15038 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250905 19:59:10.318015 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20250905 19:59:10.317982 15074 catalog_manager.cc:5582] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c reported cstate change: term changed from 3 to 4, leader changed from ac1ba6635696411790dd9f88b814fcda (127.14.175.129) to 6ad4db79bc5e462a988e8e57f6af5b6c (127.14.175.130). New cstate: current_term: 4 leader_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:10.343886 15252 raft_consensus.cc:1238] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 4 LEADER]: Rejecting Update request from peer ac1ba6635696411790dd9f88b814fcda for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:10.345013 15342 consensus_queue.cc:1046] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:10.345549 15342 consensus_queue.cc:1230] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [LEADER]: Peer 6ad4db79bc5e462a988e8e57f6af5b6c log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250905 19:59:10.345901 15333 raft_consensus.cc:3053] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:10.346130 15333 raft_consensus.cc:738] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac1ba6635696411790dd9f88b814fcda, State: Running, Role: LEADER
I20250905 19:59:10.346580 15333 consensus_queue.cc:260] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 } } peers { permanent_uuid: "6ad4db79bc5e462a988e8e57f6af5b6c" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 33675 } }
I20250905 19:59:10.347191 15333 raft_consensus.cc:3058] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:10.421397 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20250905 19:59:10.624743 15038 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20250905 19:59:10.858587 15177 raft_consensus.cc:1273] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 4 FOLLOWER]: Refusing update from remote peer 6ad4db79bc5e462a988e8e57f6af5b6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:59:10.859791 15348 consensus_queue.cc:1035] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1ba6635696411790dd9f88b814fcda" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:11.130365 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
I20250905 19:59:11.148805 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:11.149361 15038 tablet_replica.cc:331] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda: stopping tablet replica
I20250905 19:59:11.149924 15038 raft_consensus.cc:2241] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:11.150353 15038 raft_consensus.cc:2270] T b5faca015ad4488c8c2facc5f97a3ac7 P ac1ba6635696411790dd9f88b814fcda [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:11.152570 15038 tablet_replica.cc:331] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda: stopping tablet replica
I20250905 19:59:11.153005 15038 raft_consensus.cc:2241] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:59:11.153502 15038 raft_consensus.cc:2270] T 7e5ace6046144c9e91240338c6cfb93b P ac1ba6635696411790dd9f88b814fcda [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:11.234838 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 19:59:11.243654 15038 tablet_server.cc:178] TabletServer@127.14.175.130:0 shutting down...
I20250905 19:59:11.257860 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:11.258412 15038 tablet_replica.cc:331] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c: stopping tablet replica
I20250905 19:59:11.258940 15038 raft_consensus.cc:2241] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:11.259286 15038 raft_consensus.cc:2270] T 7e5ace6046144c9e91240338c6cfb93b P 6ad4db79bc5e462a988e8e57f6af5b6c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:11.260602 15038 tablet_replica.cc:331] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c: stopping tablet replica
I20250905 19:59:11.260967 15038 raft_consensus.cc:2241] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 4 LEADER]: Raft consensus shutting down.
I20250905 19:59:11.261420 15038 raft_consensus.cc:2270] T b5faca015ad4488c8c2facc5f97a3ac7 P 6ad4db79bc5e462a988e8e57f6af5b6c [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:11.269559 15038 tablet_server.cc:195] TabletServer@127.14.175.130:0 shutdown complete.
I20250905 19:59:11.277231 15038 master.cc:561] Master@127.14.175.190:38789 shutting down...
I20250905 19:59:11.392928 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:11.393407 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:11.393683 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P b9186063ede54c8cbdf48ec3d60d4b67: stopping tablet replica
I20250905 19:59:11.411080 15038 master.cc:583] Master@127.14.175.190:38789 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (6865 ms)
[ RUN ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20250905 19:59:11.442824 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:36451
I20250905 19:59:11.443819 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:11.448199 15358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:11.449338 15359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:11.451323 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:11.451618 15361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:11.452311 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:11.452500 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:11.452646 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102351452628 us; error 0 us; skew 500 ppm
I20250905 19:59:11.453138 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:11.455449 15038 webserver.cc:480] Webserver started at http://127.14.175.190:42143/ using document root <none> and password file <none>
I20250905 19:59:11.455893 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:11.456053 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:11.456306 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:11.457291 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "8b9f80d38b0140219c0208e21ea646f7"
format_stamp: "Formatted at 2025-09-05 19:59:11 on dist-test-slave-22s0"
I20250905 19:59:11.461423 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 19:59:11.464461 15366 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:11.465126 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:59:11.465368 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "8b9f80d38b0140219c0208e21ea646f7"
format_stamp: "Formatted at 2025-09-05 19:59:11 on dist-test-slave-22s0"
I20250905 19:59:11.465610 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:11.485718 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:11.486689 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:11.519348 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:36451
I20250905 19:59:11.519439 15417 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:36451 every 8 connection(s)
I20250905 19:59:11.522928 15418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:11.531947 15418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7: Bootstrap starting.
I20250905 19:59:11.535722 15418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:11.539253 15418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7: No bootstrap required, opened a new log
I20250905 19:59:11.541072 15418 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9f80d38b0140219c0208e21ea646f7" member_type: VOTER }
I20250905 19:59:11.541427 15418 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:11.541659 15418 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9f80d38b0140219c0208e21ea646f7, State: Initialized, Role: FOLLOWER
I20250905 19:59:11.542203 15418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9f80d38b0140219c0208e21ea646f7" member_type: VOTER }
I20250905 19:59:11.542644 15418 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:11.542871 15418 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:11.543124 15418 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:11.547049 15418 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9f80d38b0140219c0208e21ea646f7" member_type: VOTER }
I20250905 19:59:11.547572 15418 leader_election.cc:304] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b9f80d38b0140219c0208e21ea646f7; no voters:
I20250905 19:59:11.548593 15418 leader_election.cc:290] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:11.548849 15421 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:11.550177 15421 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [term 1 LEADER]: Becoming Leader. State: Replica: 8b9f80d38b0140219c0208e21ea646f7, State: Running, Role: LEADER
I20250905 19:59:11.550850 15421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9f80d38b0140219c0208e21ea646f7" member_type: VOTER }
I20250905 19:59:11.551410 15418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:11.553555 15422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b9f80d38b0140219c0208e21ea646f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9f80d38b0140219c0208e21ea646f7" member_type: VOTER } }
I20250905 19:59:11.553654 15423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b9f80d38b0140219c0208e21ea646f7. Latest consensus state: current_term: 1 leader_uuid: "8b9f80d38b0140219c0208e21ea646f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9f80d38b0140219c0208e21ea646f7" member_type: VOTER } }
I20250905 19:59:11.554327 15423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:11.554280 15422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:11.557654 15427 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:11.565459 15427 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:11.566298 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:59:11.573339 15427 catalog_manager.cc:1349] Generated new cluster ID: 514a800dab4d460b834adbb3a146affa
I20250905 19:59:11.573582 15427 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:11.588754 15427 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:59:11.589867 15427 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:11.601537 15427 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7: Generated new TSK 0
I20250905 19:59:11.602095 15427 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:11.632959 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:11.638166 15439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:11.639997 15440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:11.640782 15442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:11.641170 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:11.641928 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:11.642102 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:11.642218 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102351642207 us; error 0 us; skew 500 ppm
I20250905 19:59:11.642710 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:11.644831 15038 webserver.cc:480] Webserver started at http://127.14.175.129:39517/ using document root <none> and password file <none>
I20250905 19:59:11.645231 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:11.645373 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:11.645568 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:11.646615 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "a74133514c0a43a39417104e13b15c93"
format_stamp: "Formatted at 2025-09-05 19:59:11 on dist-test-slave-22s0"
I20250905 19:59:11.650558 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20250905 19:59:11.653599 15447 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:11.654271 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 19:59:11.654543 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "a74133514c0a43a39417104e13b15c93"
format_stamp: "Formatted at 2025-09-05 19:59:11 on dist-test-slave-22s0"
I20250905 19:59:11.654811 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:11.678041 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:11.679013 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:11.680275 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:11.682325 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:11.682524 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:11.682775 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:11.682917 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:11.719199 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:46193
I20250905 19:59:11.719285 15509 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:46193 every 8 connection(s)
I20250905 19:59:11.723395 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:11.730082 15514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:11.731971 15515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:11.733675 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:11.733750 15517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:11.734647 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:11.734861 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:11.735041 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102351735023 us; error 0 us; skew 500 ppm
I20250905 19:59:11.735595 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:11.738257 15038 webserver.cc:480] Webserver started at http://127.14.175.130:44339/ using document root <none> and password file <none>
I20250905 19:59:11.738291 15510 heartbeater.cc:344] Connected to a master server at 127.14.175.190:36451
I20250905 19:59:11.738651 15510 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:11.738852 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:11.739061 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:11.739334 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:11.739384 15510 heartbeater.cc:507] Master 127.14.175.190:36451 requested a full tablet report, sending...
I20250905 19:59:11.740798 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/ts-1-root/instance:
uuid: "3131e44438f841faaf84894cf7bff667"
format_stamp: "Formatted at 2025-09-05 19:59:11 on dist-test-slave-22s0"
I20250905 19:59:11.741384 15383 ts_manager.cc:194] Registered new tserver with Master: a74133514c0a43a39417104e13b15c93 (127.14.175.129:46193)
I20250905 19:59:11.743013 15383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40974
I20250905 19:59:11.745322 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20250905 19:59:11.748386 15522 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:11.749044 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:59:11.749300 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/ts-1-root
uuid: "3131e44438f841faaf84894cf7bff667"
format_stamp: "Formatted at 2025-09-05 19:59:11 on dist-test-slave-22s0"
I20250905 19:59:11.749553 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1757102344413625-15038-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:11.759325 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:11.760332 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:11.761652 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:11.763717 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:11.763906 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:11.764137 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:11.764287 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:11.800295 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.130:33243
I20250905 19:59:11.800391 15584 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.130:33243 every 8 connection(s)
I20250905 19:59:11.812285 15585 heartbeater.cc:344] Connected to a master server at 127.14.175.190:36451
I20250905 19:59:11.812599 15585 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:11.813215 15585 heartbeater.cc:507] Master 127.14.175.190:36451 requested a full tablet report, sending...
I20250905 19:59:11.814793 15383 ts_manager.cc:194] Registered new tserver with Master: 3131e44438f841faaf84894cf7bff667 (127.14.175.130:33243)
I20250905 19:59:11.815169 15038 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.011910957s
I20250905 19:59:11.816291 15383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40990
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:422: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20250905 19:59:11.835876 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
I20250905 19:59:11.845301 15510 heartbeater.cc:499] Master 127.14.175.190:36451 was elected leader, sending a full tablet report...
I20250905 19:59:11.852725 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:11.867601 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 19:59:11.873790 15038 tablet_server.cc:178] TabletServer@127.14.175.130:0 shutting down...
I20250905 19:59:11.887270 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:11.901997 15038 tablet_server.cc:195] TabletServer@127.14.175.130:0 shutdown complete.
I20250905 19:59:11.908133 15038 master.cc:561] Master@127.14.175.190:36451 shutting down...
I20250905 19:59:11.920471 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:11.920939 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:11.921303 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P 8b9f80d38b0140219c0208e21ea646f7: stopping tablet replica
I20250905 19:59:11.929224 15038 master.cc:583] Master@127.14.175.190:36451 shutdown complete.
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (507 ms)
[ RUN ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20250905 19:59:11.949911 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:42359
I20250905 19:59:11.950946 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:11.955432 15598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:11.956586 15599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:11.957719 15601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:11.958279 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:11.958947 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:11.959112 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:11.959226 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102351959215 us; error 0 us; skew 500 ppm
I20250905 19:59:11.959650 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:11.961712 15038 webserver.cc:480] Webserver started at http://127.14.175.190:33965/ using document root <none> and password file <none>
I20250905 19:59:11.962131 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:11.962275 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:11.962487 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:11.963568 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "f09466a91a60411b845071eafa0e4a57"
format_stamp: "Formatted at 2025-09-05 19:59:11 on dist-test-slave-22s0"
I20250905 19:59:11.967592 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250905 19:59:11.970599 15606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:11.971297 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 19:59:11.971558 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "f09466a91a60411b845071eafa0e4a57"
format_stamp: "Formatted at 2025-09-05 19:59:11 on dist-test-slave-22s0"
I20250905 19:59:11.971822 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:12.009867 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:12.010866 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:12.043594 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:42359
I20250905 19:59:12.043700 15657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:42359 every 8 connection(s)
I20250905 19:59:12.047137 15658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:12.056471 15658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57: Bootstrap starting.
I20250905 19:59:12.060384 15658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:12.063913 15658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57: No bootstrap required, opened a new log
I20250905 19:59:12.065610 15658 raft_consensus.cc:357] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09466a91a60411b845071eafa0e4a57" member_type: VOTER }
I20250905 19:59:12.065961 15658 raft_consensus.cc:383] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:12.066133 15658 raft_consensus.cc:738] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09466a91a60411b845071eafa0e4a57, State: Initialized, Role: FOLLOWER
I20250905 19:59:12.066634 15658 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09466a91a60411b845071eafa0e4a57" member_type: VOTER }
I20250905 19:59:12.067021 15658 raft_consensus.cc:397] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:12.067203 15658 raft_consensus.cc:491] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:12.067404 15658 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:12.071290 15658 raft_consensus.cc:513] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09466a91a60411b845071eafa0e4a57" member_type: VOTER }
I20250905 19:59:12.071790 15658 leader_election.cc:304] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f09466a91a60411b845071eafa0e4a57; no voters:
I20250905 19:59:12.072821 15658 leader_election.cc:290] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:12.073083 15661 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:12.074481 15661 raft_consensus.cc:695] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [term 1 LEADER]: Becoming Leader. State: Replica: f09466a91a60411b845071eafa0e4a57, State: Running, Role: LEADER
I20250905 19:59:12.075150 15661 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09466a91a60411b845071eafa0e4a57" member_type: VOTER }
I20250905 19:59:12.075712 15658 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:12.077833 15662 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f09466a91a60411b845071eafa0e4a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09466a91a60411b845071eafa0e4a57" member_type: VOTER } }
I20250905 19:59:12.077895 15663 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09466a91a60411b845071eafa0e4a57. Latest consensus state: current_term: 1 leader_uuid: "f09466a91a60411b845071eafa0e4a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09466a91a60411b845071eafa0e4a57" member_type: VOTER } }
I20250905 19:59:12.078572 15663 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:12.078629 15662 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:12.082461 15666 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:12.086794 15666 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:12.091367 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:59:12.094777 15666 catalog_manager.cc:1349] Generated new cluster ID: 3c7b1dead45e45e48e6772dabca7dc76
I20250905 19:59:12.095034 15666 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:12.125654 15666 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:59:12.126855 15666 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:12.139147 15666 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57: Generated new TSK 0
I20250905 19:59:12.139696 15666 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:12.158002 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:12.163630 15679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:12.164062 15680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:12.166224 15682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:12.166361 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:12.167235 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:12.167411 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:12.167539 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102352167528 us; error 0 us; skew 500 ppm
I20250905 19:59:12.167951 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:12.170069 15038 webserver.cc:480] Webserver started at http://127.14.175.129:41275/ using document root <none> and password file <none>
I20250905 19:59:12.170529 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:12.170706 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:12.170913 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:12.171885 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "5db0c1906cae4295a21f7fcfcf3c0954"
format_stamp: "Formatted at 2025-09-05 19:59:12 on dist-test-slave-22s0"
I20250905 19:59:12.176110 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 19:59:12.179351 15687 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:12.180040 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 19:59:12.180284 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "5db0c1906cae4295a21f7fcfcf3c0954"
format_stamp: "Formatted at 2025-09-05 19:59:12 on dist-test-slave-22s0"
I20250905 19:59:12.180550 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:12.197643 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:12.198632 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:12.199913 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:12.201970 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:12.202142 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:12.202396 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:12.202560 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250905 19:59:12.237088 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:38047
I20250905 19:59:12.237175 15749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:38047 every 8 connection(s)
I20250905 19:59:12.241365 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:12.248067 15754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:12.249276 15755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:12.255252 15750 heartbeater.cc:344] Connected to a master server at 127.14.175.190:42359
I20250905 19:59:12.255661 15750 heartbeater.cc:461] Registering TS with master...
W20250905 19:59:12.256188 15757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:12.256461 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:12.256695 15750 heartbeater.cc:507] Master 127.14.175.190:42359 requested a full tablet report, sending...
I20250905 19:59:12.257432 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:12.257695 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:12.257879 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102352257860 us; error 0 us; skew 500 ppm
I20250905 19:59:12.258545 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:12.258848 15623 ts_manager.cc:194] Registered new tserver with Master: 5db0c1906cae4295a21f7fcfcf3c0954 (127.14.175.129:38047)
I20250905 19:59:12.260527 15623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33944
I20250905 19:59:12.261283 15038 webserver.cc:480] Webserver started at http://127.14.175.130:39127/ using document root <none> and password file <none>
I20250905 19:59:12.261919 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:12.262159 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:12.262471 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:12.263540 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/ts-1-root/instance:
uuid: "e5bd8768f1c340d98dda2b91d8d7fda1"
format_stamp: "Formatted at 2025-09-05 19:59:12 on dist-test-slave-22s0"
I20250905 19:59:12.267539 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250905 19:59:12.270447 15762 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:12.271150 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 19:59:12.271404 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/ts-1-root
uuid: "e5bd8768f1c340d98dda2b91d8d7fda1"
format_stamp: "Formatted at 2025-09-05 19:59:12 on dist-test-slave-22s0"
I20250905 19:59:12.271643 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1757102344413625-15038-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:12.290459 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:12.291410 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:12.292742 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:12.294768 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:12.294960 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:12.295187 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:12.295341 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:12.330085 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.130:45903
I20250905 19:59:12.330175 15824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.130:45903 every 8 connection(s)
I20250905 19:59:12.341972 15825 heartbeater.cc:344] Connected to a master server at 127.14.175.190:42359
I20250905 19:59:12.342298 15825 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:12.342960 15825 heartbeater.cc:507] Master 127.14.175.190:42359 requested a full tablet report, sending...
I20250905 19:59:12.344650 15623 ts_manager.cc:194] Registered new tserver with Master: e5bd8768f1c340d98dda2b91d8d7fda1 (127.14.175.130:45903)
I20250905 19:59:12.345108 15038 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.011943356s
I20250905 19:59:12.345891 15623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33950
I20250905 19:59:12.362860 15750 heartbeater.cc:499] Master 127.14.175.190:42359 was elected leader, sending a full tablet report...
I20250905 19:59:12.365687 15623 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:59:12.367034 15623 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20250905 19:59:12.395885 15790 tablet_service.cc:1468] Processing CreateTablet for tablet a4d6ad80b1194d3cb7b0219b2c8b34f4 (DEFAULT_TABLE table=test-table [id=a2a344d4706a45d89dd944e778481ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:12.397001 15790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d6ad80b1194d3cb7b0219b2c8b34f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:12.397228 15715 tablet_service.cc:1468] Processing CreateTablet for tablet a4d6ad80b1194d3cb7b0219b2c8b34f4 (DEFAULT_TABLE table=test-table [id=a2a344d4706a45d89dd944e778481ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:12.398114 15715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d6ad80b1194d3cb7b0219b2c8b34f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:12.408551 15838 tablet_bootstrap.cc:492] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1: Bootstrap starting.
I20250905 19:59:12.409768 15839 tablet_bootstrap.cc:492] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954: Bootstrap starting.
I20250905 19:59:12.413187 15838 tablet_bootstrap.cc:654] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:12.414420 15839 tablet_bootstrap.cc:654] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:12.417157 15838 tablet_bootstrap.cc:492] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1: No bootstrap required, opened a new log
I20250905 19:59:12.417482 15838 ts_tablet_manager.cc:1397] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20250905 19:59:12.418382 15839 tablet_bootstrap.cc:492] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954: No bootstrap required, opened a new log
I20250905 19:59:12.418762 15839 ts_tablet_manager.cc:1397] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.001s
I20250905 19:59:12.419279 15838 raft_consensus.cc:357] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bd8768f1c340d98dda2b91d8d7fda1" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45903 } } peers { permanent_uuid: "5db0c1906cae4295a21f7fcfcf3c0954" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 38047 } }
I20250905 19:59:12.419730 15838 raft_consensus.cc:738] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5bd8768f1c340d98dda2b91d8d7fda1, State: Initialized, Role: FOLLOWER
I20250905 19:59:12.420172 15838 consensus_queue.cc:260] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bd8768f1c340d98dda2b91d8d7fda1" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45903 } } peers { permanent_uuid: "5db0c1906cae4295a21f7fcfcf3c0954" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 38047 } }
I20250905 19:59:12.420445 15839 raft_consensus.cc:357] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bd8768f1c340d98dda2b91d8d7fda1" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45903 } } peers { permanent_uuid: "5db0c1906cae4295a21f7fcfcf3c0954" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 38047 } }
I20250905 19:59:12.421041 15839 raft_consensus.cc:738] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db0c1906cae4295a21f7fcfcf3c0954, State: Initialized, Role: FOLLOWER
I20250905 19:59:12.421540 15839 consensus_queue.cc:260] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bd8768f1c340d98dda2b91d8d7fda1" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45903 } } peers { permanent_uuid: "5db0c1906cae4295a21f7fcfcf3c0954" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 38047 } }
I20250905 19:59:12.423027 15825 heartbeater.cc:499] Master 127.14.175.190:42359 was elected leader, sending a full tablet report...
I20250905 19:59:12.423633 15838 ts_tablet_manager.cc:1428] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20250905 19:59:12.428114 15839 ts_tablet_manager.cc:1428] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954: Time spent starting tablet: real 0.009s user 0.000s sys 0.008s
I20250905 19:59:12.458266 15038 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20250905 19:59:12.458496 15038 raft_consensus.cc:421] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 0 FOLLOWER]: Emulating election...
I20250905 19:59:12.458791 15038 raft_consensus.cc:3058] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:12.463841 15038 raft_consensus.cc:695] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 1 LEADER]: Becoming Leader. State: Replica: 5db0c1906cae4295a21f7fcfcf3c0954, State: Running, Role: LEADER
W20250905 19:59:12.464154 15750 heartbeater.cc:646] Failed to heartbeat to 127.14.175.190:42359 (0 consecutive failures): IO error: failing all heartbeats for tests
I20250905 19:59:12.464414 15038 consensus_queue.cc:237] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bd8768f1c340d98dda2b91d8d7fda1" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45903 } } peers { permanent_uuid: "5db0c1906cae4295a21f7fcfcf3c0954" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 38047 } }
I20250905 19:59:12.467080 15038 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250905 19:59:12.484297 15038 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0
I20250905 19:59:12.588001 15038 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0
I20250905 19:59:12.792032 15038 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0
I20250905 19:59:12.899823 15800 raft_consensus.cc:3058] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:12.905320 15843 consensus_queue.cc:1035] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd8768f1c340d98dda2b91d8d7fda1" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:59:13.096827 15038 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20250905 19:59:13.097066 15038 raft_consensus.cc:421] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 1 LEADER]: Emulating election...
I20250905 19:59:13.097285 15038 raft_consensus.cc:3053] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:13.097472 15038 raft_consensus.cc:738] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5db0c1906cae4295a21f7fcfcf3c0954, State: Running, Role: LEADER
I20250905 19:59:13.097937 15038 consensus_queue.cc:260] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bd8768f1c340d98dda2b91d8d7fda1" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45903 } } peers { permanent_uuid: "5db0c1906cae4295a21f7fcfcf3c0954" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 38047 } }
I20250905 19:59:13.098587 15038 raft_consensus.cc:3058] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:13.102476 15038 raft_consensus.cc:695] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 2 LEADER]: Becoming Leader. State: Replica: 5db0c1906cae4295a21f7fcfcf3c0954, State: Running, Role: LEADER
I20250905 19:59:13.103053 15038 consensus_queue.cc:237] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bd8768f1c340d98dda2b91d8d7fda1" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45903 } } peers { permanent_uuid: "5db0c1906cae4295a21f7fcfcf3c0954" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 38047 } }
I20250905 19:59:13.104322 15038 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250905 19:59:13.107798 15038 cluster_itest_util.cc:257] Not converged past 2 yet: 2.2 1.1
I20250905 19:59:13.211687 15038 cluster_itest_util.cc:257] Not converged past 2 yet: 2.2 1.1
I20250905 19:59:13.416193 15038 cluster_itest_util.cc:257] Not converged past 2 yet: 2.2 1.1
I20250905 19:59:13.649292 15800 raft_consensus.cc:3058] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:13.653867 15800 raft_consensus.cc:1273] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1 [term 2 FOLLOWER]: Refusing update from remote peer 5db0c1906cae4295a21f7fcfcf3c0954: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:13.655028 15851 consensus_queue.cc:1035] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd8768f1c340d98dda2b91d8d7fda1" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:13.721169 15038 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20250905 19:59:13.721400 15038 raft_consensus.cc:421] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 2 LEADER]: Emulating election...
I20250905 19:59:13.721593 15038 raft_consensus.cc:3053] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:13.721743 15038 raft_consensus.cc:738] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5db0c1906cae4295a21f7fcfcf3c0954, State: Running, Role: LEADER
I20250905 19:59:13.722108 15038 consensus_queue.cc:260] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bd8768f1c340d98dda2b91d8d7fda1" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45903 } } peers { permanent_uuid: "5db0c1906cae4295a21f7fcfcf3c0954" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 38047 } }
I20250905 19:59:13.722699 15038 raft_consensus.cc:3058] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:13.726397 15038 raft_consensus.cc:695] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 3 LEADER]: Becoming Leader. State: Replica: 5db0c1906cae4295a21f7fcfcf3c0954, State: Running, Role: LEADER
I20250905 19:59:13.726999 15038 consensus_queue.cc:237] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bd8768f1c340d98dda2b91d8d7fda1" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45903 } } peers { permanent_uuid: "5db0c1906cae4295a21f7fcfcf3c0954" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 38047 } }
I20250905 19:59:13.728201 15038 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250905 19:59:13.731266 15038 cluster_itest_util.cc:257] Not converged past 3 yet: 3.3 2.2
I20250905 19:59:13.835204 15038 cluster_itest_util.cc:257] Not converged past 3 yet: 3.3 2.2
I20250905 19:59:14.039189 15038 cluster_itest_util.cc:257] Not converged past 3 yet: 3.3 2.2
I20250905 19:59:14.150029 15800 raft_consensus.cc:3058] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:14.154536 15800 raft_consensus.cc:1273] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1 [term 3 FOLLOWER]: Refusing update from remote peer 5db0c1906cae4295a21f7fcfcf3c0954: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:14.155756 15853 consensus_queue.cc:1035] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd8768f1c340d98dda2b91d8d7fda1" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:14.346205 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
I20250905 19:59:14.359615 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:14.360133 15038 tablet_replica.cc:331] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954: stopping tablet replica
I20250905 19:59:14.360615 15038 raft_consensus.cc:2241] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 3 LEADER]: Raft consensus shutting down.
I20250905 19:59:14.361212 15038 raft_consensus.cc:2270] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P 5db0c1906cae4295a21f7fcfcf3c0954 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:14.379464 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 19:59:14.386886 15038 tablet_server.cc:178] TabletServer@127.14.175.130:0 shutting down...
I20250905 19:59:14.400629 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:14.401103 15038 tablet_replica.cc:331] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1: stopping tablet replica
I20250905 19:59:14.401508 15038 raft_consensus.cc:2241] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:14.401875 15038 raft_consensus.cc:2270] T a4d6ad80b1194d3cb7b0219b2c8b34f4 P e5bd8768f1c340d98dda2b91d8d7fda1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:14.419191 15038 tablet_server.cc:195] TabletServer@127.14.175.130:0 shutdown complete.
I20250905 19:59:14.426038 15038 master.cc:561] Master@127.14.175.190:42359 shutting down...
I20250905 19:59:14.438700 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:14.439165 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:14.439502 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P f09466a91a60411b845071eafa0e4a57: stopping tablet replica
I20250905 19:59:14.455698 15038 master.cc:583] Master@127.14.175.190:42359 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (2529 ms)
[----------] 3 tests from LeadershipChangeReportingTest (9903 ms total)
[----------] 4 tests from TsTabletManagerITest
[ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20250905 19:59:14.480592 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:33213
I20250905 19:59:14.481563 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:14.486181 15861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:14.487437 15862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:14.489076 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:14.489442 15864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:14.490119 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:14.490306 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:14.490449 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102354490433 us; error 0 us; skew 500 ppm
I20250905 19:59:14.490983 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:14.493204 15038 webserver.cc:480] Webserver started at http://127.14.175.190:46471/ using document root <none> and password file <none>
I20250905 19:59:14.493666 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:14.493840 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:14.494074 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:14.495108 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "c20fa7fd2841421695928da3e69f177b"
format_stamp: "Formatted at 2025-09-05 19:59:14 on dist-test-slave-22s0"
I20250905 19:59:14.499164 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250905 19:59:14.502149 15869 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:14.502959 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 19:59:14.503415 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "c20fa7fd2841421695928da3e69f177b"
format_stamp: "Formatted at 2025-09-05 19:59:14 on dist-test-slave-22s0"
I20250905 19:59:14.503950 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:14.517910 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:14.518857 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:14.552132 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:33213
I20250905 19:59:14.552219 15920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:33213 every 8 connection(s)
I20250905 19:59:14.555630 15921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:14.564787 15921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b: Bootstrap starting.
I20250905 19:59:14.568696 15921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:14.572155 15921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b: No bootstrap required, opened a new log
I20250905 19:59:14.573822 15921 raft_consensus.cc:357] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20fa7fd2841421695928da3e69f177b" member_type: VOTER }
I20250905 19:59:14.574159 15921 raft_consensus.cc:383] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:14.574334 15921 raft_consensus.cc:738] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20fa7fd2841421695928da3e69f177b, State: Initialized, Role: FOLLOWER
I20250905 19:59:14.574820 15921 consensus_queue.cc:260] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20fa7fd2841421695928da3e69f177b" member_type: VOTER }
I20250905 19:59:14.575205 15921 raft_consensus.cc:397] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:14.575384 15921 raft_consensus.cc:491] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:14.575585 15921 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:14.579525 15921 raft_consensus.cc:513] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20fa7fd2841421695928da3e69f177b" member_type: VOTER }
I20250905 19:59:14.579964 15921 leader_election.cc:304] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c20fa7fd2841421695928da3e69f177b; no voters:
I20250905 19:59:14.580930 15921 leader_election.cc:290] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:14.581189 15924 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:14.582454 15924 raft_consensus.cc:695] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [term 1 LEADER]: Becoming Leader. State: Replica: c20fa7fd2841421695928da3e69f177b, State: Running, Role: LEADER
I20250905 19:59:14.583207 15924 consensus_queue.cc:237] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20fa7fd2841421695928da3e69f177b" member_type: VOTER }
I20250905 19:59:14.583875 15921 sys_catalog.cc:564] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:14.586104 15925 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c20fa7fd2841421695928da3e69f177b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20fa7fd2841421695928da3e69f177b" member_type: VOTER } }
I20250905 19:59:14.587255 15925 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:14.587436 15926 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c20fa7fd2841421695928da3e69f177b. Latest consensus state: current_term: 1 leader_uuid: "c20fa7fd2841421695928da3e69f177b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20fa7fd2841421695928da3e69f177b" member_type: VOTER } }
I20250905 19:59:14.602586 15926 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:14.606140 15929 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:14.613224 15929 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:14.614670 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:59:14.622282 15929 catalog_manager.cc:1349] Generated new cluster ID: 8663a803bd634a488f48fd739150e230
I20250905 19:59:14.622622 15929 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:14.642225 15929 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:59:14.643509 15929 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:14.654011 15929 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b: Generated new TSK 0
I20250905 19:59:14.654636 15929 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:14.681473 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:14.688758 15943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:14.689674 15944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:14.691507 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:14.691618 15946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:14.692598 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:14.692793 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:14.692917 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102354692905 us; error 0 us; skew 500 ppm
I20250905 19:59:14.693408 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:14.695964 15038 webserver.cc:480] Webserver started at http://127.14.175.129:33195/ using document root <none> and password file <none>
I20250905 19:59:14.696439 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:14.696636 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:14.696882 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:14.698081 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "84d3dcd725c64c339b1f0dcc03e5a605"
format_stamp: "Formatted at 2025-09-05 19:59:14 on dist-test-slave-22s0"
I20250905 19:59:14.702620 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250905 19:59:14.705982 15951 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:14.706843 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 19:59:14.707118 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "84d3dcd725c64c339b1f0dcc03e5a605"
format_stamp: "Formatted at 2025-09-05 19:59:14 on dist-test-slave-22s0"
I20250905 19:59:14.707379 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:14.727507 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:14.728793 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:14.730324 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:14.732674 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:14.732901 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:14.733131 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:14.733291 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:14.773497 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:39405
I20250905 19:59:14.773598 16013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:39405 every 8 connection(s)
I20250905 19:59:14.778339 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:14.785786 16018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:14.787588 16019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:14.791356 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:14.792194 16021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:14.793067 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:14.793324 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:14.793334 16014 heartbeater.cc:344] Connected to a master server at 127.14.175.190:33213
I20250905 19:59:14.793495 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102354793479 us; error 0 us; skew 500 ppm
I20250905 19:59:14.793776 16014 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:14.794118 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:14.794479 16014 heartbeater.cc:507] Master 127.14.175.190:33213 requested a full tablet report, sending...
I20250905 19:59:14.796720 15038 webserver.cc:480] Webserver started at http://127.14.175.130:34847/ using document root <none> and password file <none>
I20250905 19:59:14.796697 15886 ts_manager.cc:194] Registered new tserver with Master: 84d3dcd725c64c339b1f0dcc03e5a605 (127.14.175.129:39405)
I20250905 19:59:14.797281 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:14.797474 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:14.797719 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:14.798606 15886 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34464
I20250905 19:59:14.798894 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-1-root/instance:
uuid: "f67dee448000427bacbf8414c3a5c134"
format_stamp: "Formatted at 2025-09-05 19:59:14 on dist-test-slave-22s0"
I20250905 19:59:14.803185 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250905 19:59:14.806321 16026 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:14.807060 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 19:59:14.807313 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-1-root
uuid: "f67dee448000427bacbf8414c3a5c134"
format_stamp: "Formatted at 2025-09-05 19:59:14 on dist-test-slave-22s0"
I20250905 19:59:14.807583 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:14.816468 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:14.817541 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:14.818943 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:14.820996 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:14.821194 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:14.821415 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:14.821573 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:14.860158 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.130:35429
I20250905 19:59:14.860241 16088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.130:35429 every 8 connection(s)
I20250905 19:59:14.864662 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:14.871030 16092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:14.872735 16093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:14.874925 16095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:14.875456 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:14.876452 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:14.876688 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:14.876897 16089 heartbeater.cc:344] Connected to a master server at 127.14.175.190:33213
I20250905 19:59:14.876960 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102354876935 us; error 0 us; skew 500 ppm
I20250905 19:59:14.877234 16089 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:14.877537 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:14.877944 16089 heartbeater.cc:507] Master 127.14.175.190:33213 requested a full tablet report, sending...
I20250905 19:59:14.879964 15886 ts_manager.cc:194] Registered new tserver with Master: f67dee448000427bacbf8414c3a5c134 (127.14.175.130:35429)
I20250905 19:59:14.880156 15038 webserver.cc:480] Webserver started at http://127.14.175.131:34563/ using document root <none> and password file <none>
I20250905 19:59:14.880666 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:14.880846 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:14.881088 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:14.881387 15886 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34466
I20250905 19:59:14.882324 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-2-root/instance:
uuid: "37dcf68961894c1d9d0e691812799a37"
format_stamp: "Formatted at 2025-09-05 19:59:14 on dist-test-slave-22s0"
I20250905 19:59:14.886899 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250905 19:59:14.890055 16100 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:14.890761 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 19:59:14.891033 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-2-root
uuid: "37dcf68961894c1d9d0e691812799a37"
format_stamp: "Formatted at 2025-09-05 19:59:14 on dist-test-slave-22s0"
I20250905 19:59:14.891302 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1757102344413625-15038-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:14.905431 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:14.906672 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:14.908080 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:14.910532 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:14.910741 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:14.910975 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:14.911132 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:14.980998 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.131:40723
I20250905 19:59:14.981096 16162 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.131:40723 every 8 connection(s)
I20250905 19:59:14.996224 16163 heartbeater.cc:344] Connected to a master server at 127.14.175.190:33213
I20250905 19:59:14.996569 16163 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:14.997323 16163 heartbeater.cc:507] Master 127.14.175.190:33213 requested a full tablet report, sending...
I20250905 19:59:14.999204 15886 ts_manager.cc:194] Registered new tserver with Master: 37dcf68961894c1d9d0e691812799a37 (127.14.175.131:40723)
I20250905 19:59:14.999545 15038 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014522569s
I20250905 19:59:15.000999 15886 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34482
I20250905 19:59:15.025296 15886 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20250905 19:59:15.026873 15886 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:59:15.068336 15979 tablet_service.cc:1468] Processing CreateTablet for tablet 78a05c6028354e418c6ee7256bef1491 (DEFAULT_TABLE table=test-table [id=fb29e66908364c72af59935f99c09a2c]), partition=
I20250905 19:59:15.069605 15979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a05c6028354e418c6ee7256bef1491. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:15.070327 16054 tablet_service.cc:1468] Processing CreateTablet for tablet 78a05c6028354e418c6ee7256bef1491 (DEFAULT_TABLE table=test-table [id=fb29e66908364c72af59935f99c09a2c]), partition=
I20250905 19:59:15.071355 16054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a05c6028354e418c6ee7256bef1491. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:15.072683 16128 tablet_service.cc:1468] Processing CreateTablet for tablet 78a05c6028354e418c6ee7256bef1491 (DEFAULT_TABLE table=test-table [id=fb29e66908364c72af59935f99c09a2c]), partition=
I20250905 19:59:15.073499 16128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a05c6028354e418c6ee7256bef1491. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:15.087965 16178 tablet_bootstrap.cc:492] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37: Bootstrap starting.
I20250905 19:59:15.089286 16180 tablet_bootstrap.cc:492] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605: Bootstrap starting.
I20250905 19:59:15.090901 16179 tablet_bootstrap.cc:492] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134: Bootstrap starting.
I20250905 19:59:15.095912 16180 tablet_bootstrap.cc:654] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:15.097373 16178 tablet_bootstrap.cc:654] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:15.098050 16179 tablet_bootstrap.cc:654] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:15.106848 16179 tablet_bootstrap.cc:492] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134: No bootstrap required, opened a new log
I20250905 19:59:15.106981 16180 tablet_bootstrap.cc:492] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605: No bootstrap required, opened a new log
I20250905 19:59:15.107275 16179 ts_tablet_manager.cc:1397] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20250905 19:59:15.107443 16180 ts_tablet_manager.cc:1397] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.010s
I20250905 19:59:15.107546 16178 tablet_bootstrap.cc:492] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37: No bootstrap required, opened a new log
I20250905 19:59:15.108081 16178 ts_tablet_manager.cc:1397] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.009s
I20250905 19:59:15.110433 16179 raft_consensus.cc:357] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:15.110700 16180 raft_consensus.cc:357] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:15.110999 16178 raft_consensus.cc:357] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:15.111373 16179 raft_consensus.cc:383] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:15.111605 16180 raft_consensus.cc:383] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:15.111753 16178 raft_consensus.cc:383] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:15.111846 16179 raft_consensus.cc:738] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67dee448000427bacbf8414c3a5c134, State: Initialized, Role: FOLLOWER
I20250905 19:59:15.111996 16180 raft_consensus.cc:738] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3dcd725c64c339b1f0dcc03e5a605, State: Initialized, Role: FOLLOWER
I20250905 19:59:15.112167 16178 raft_consensus.cc:738] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dcf68961894c1d9d0e691812799a37, State: Initialized, Role: FOLLOWER
I20250905 19:59:15.112684 16179 consensus_queue.cc:260] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:15.112972 16178 consensus_queue.cc:260] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:15.112922 16180 consensus_queue.cc:260] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:15.115274 16089 heartbeater.cc:499] Master 127.14.175.190:33213 was elected leader, sending a full tablet report...
I20250905 19:59:15.115427 16179 ts_tablet_manager.cc:1428] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250905 19:59:15.119771 16163 heartbeater.cc:499] Master 127.14.175.190:33213 was elected leader, sending a full tablet report...
I20250905 19:59:15.120270 16178 ts_tablet_manager.cc:1428] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20250905 19:59:15.127908 16014 heartbeater.cc:499] Master 127.14.175.190:33213 was elected leader, sending a full tablet report...
I20250905 19:59:15.128594 16180 ts_tablet_manager.cc:1428] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605: Time spent starting tablet: real 0.021s user 0.007s sys 0.010s
I20250905 19:59:15.216250 16186 raft_consensus.cc:491] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:15.216723 16186 raft_consensus.cc:513] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:15.218737 16186 leader_election.cc:290] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f67dee448000427bacbf8414c3a5c134 (127.14.175.130:35429), 37dcf68961894c1d9d0e691812799a37 (127.14.175.131:40723)
I20250905 19:59:15.228876 16064 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a05c6028354e418c6ee7256bef1491" candidate_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f67dee448000427bacbf8414c3a5c134" is_pre_election: true
I20250905 19:59:15.229647 16138 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a05c6028354e418c6ee7256bef1491" candidate_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dcf68961894c1d9d0e691812799a37" is_pre_election: true
I20250905 19:59:15.229753 16064 raft_consensus.cc:2466] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3dcd725c64c339b1f0dcc03e5a605 in term 0.
I20250905 19:59:15.230376 16138 raft_consensus.cc:2466] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3dcd725c64c339b1f0dcc03e5a605 in term 0.
I20250905 19:59:15.231225 15955 leader_election.cc:304] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3dcd725c64c339b1f0dcc03e5a605, f67dee448000427bacbf8414c3a5c134; no voters:
I20250905 19:59:15.232024 16186 raft_consensus.cc:2802] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:59:15.232384 16186 raft_consensus.cc:491] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:15.232661 16186 raft_consensus.cc:3058] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:15.237195 16186 raft_consensus.cc:513] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:15.238714 16186 leader_election.cc:290] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [CANDIDATE]: Term 1 election: Requested vote from peers f67dee448000427bacbf8414c3a5c134 (127.14.175.130:35429), 37dcf68961894c1d9d0e691812799a37 (127.14.175.131:40723)
I20250905 19:59:15.239460 16064 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a05c6028354e418c6ee7256bef1491" candidate_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f67dee448000427bacbf8414c3a5c134"
I20250905 19:59:15.239741 16138 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a05c6028354e418c6ee7256bef1491" candidate_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dcf68961894c1d9d0e691812799a37"
I20250905 19:59:15.240015 16064 raft_consensus.cc:3058] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:15.240257 16138 raft_consensus.cc:3058] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:15.247341 16064 raft_consensus.cc:2466] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3dcd725c64c339b1f0dcc03e5a605 in term 1.
I20250905 19:59:15.247345 16138 raft_consensus.cc:2466] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3dcd725c64c339b1f0dcc03e5a605 in term 1.
I20250905 19:59:15.248484 15955 leader_election.cc:304] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3dcd725c64c339b1f0dcc03e5a605, f67dee448000427bacbf8414c3a5c134; no voters:
I20250905 19:59:15.249130 16186 raft_consensus.cc:2802] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:15.249938 16186 raft_consensus.cc:695] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 1 LEADER]: Becoming Leader. State: Replica: 84d3dcd725c64c339b1f0dcc03e5a605, State: Running, Role: LEADER
I20250905 19:59:15.250656 16186 consensus_queue.cc:237] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:15.256420 15886 catalog_manager.cc:5582] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84d3dcd725c64c339b1f0dcc03e5a605 (127.14.175.129). New cstate: current_term: 1 leader_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:15.669317 16186 consensus_queue.cc:1035] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:59:15.679275 16188 consensus_queue.cc:1035] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:59:16.321291 15038 tablet_replica.cc:331] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605: stopping tablet replica
I20250905 19:59:16.321808 15038 raft_consensus.cc:2241] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:16.322458 15038 raft_consensus.cc:2270] T 78a05c6028354e418c6ee7256bef1491 P 84d3dcd725c64c339b1f0dcc03e5a605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:17.581492 16199 raft_consensus.cc:491] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84d3dcd725c64c339b1f0dcc03e5a605)
I20250905 19:59:17.581946 16199 raft_consensus.cc:513] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:17.583794 16199 leader_election.cc:290] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84d3dcd725c64c339b1f0dcc03e5a605 (127.14.175.129:39405), f67dee448000427bacbf8414c3a5c134 (127.14.175.130:35429)
I20250905 19:59:17.596591 16064 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a05c6028354e418c6ee7256bef1491" candidate_uuid: "37dcf68961894c1d9d0e691812799a37" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f67dee448000427bacbf8414c3a5c134" is_pre_election: true
I20250905 19:59:17.596603 15989 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a05c6028354e418c6ee7256bef1491" candidate_uuid: "37dcf68961894c1d9d0e691812799a37" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" is_pre_election: true
W20250905 19:59:17.598031 16102 leader_election.cc:343] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84d3dcd725c64c339b1f0dcc03e5a605 (127.14.175.129:39405): Illegal state: cannot vote while shut down
I20250905 19:59:17.598404 16102 leader_election.cc:304] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37dcf68961894c1d9d0e691812799a37; no voters: 84d3dcd725c64c339b1f0dcc03e5a605, f67dee448000427bacbf8414c3a5c134
I20250905 19:59:17.599013 16199 raft_consensus.cc:2747] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:17.670841 16204 raft_consensus.cc:491] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84d3dcd725c64c339b1f0dcc03e5a605)
I20250905 19:59:17.671324 16204 raft_consensus.cc:513] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:17.673228 16204 leader_election.cc:290] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84d3dcd725c64c339b1f0dcc03e5a605 (127.14.175.129:39405), 37dcf68961894c1d9d0e691812799a37 (127.14.175.131:40723)
I20250905 19:59:17.683938 15989 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a05c6028354e418c6ee7256bef1491" candidate_uuid: "f67dee448000427bacbf8414c3a5c134" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" is_pre_election: true
W20250905 19:59:17.685144 16028 leader_election.cc:343] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84d3dcd725c64c339b1f0dcc03e5a605 (127.14.175.129:39405): Illegal state: cannot vote while shut down
I20250905 19:59:17.685204 16138 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a05c6028354e418c6ee7256bef1491" candidate_uuid: "f67dee448000427bacbf8414c3a5c134" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37dcf68961894c1d9d0e691812799a37" is_pre_election: true
I20250905 19:59:17.685865 16138 raft_consensus.cc:2466] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f67dee448000427bacbf8414c3a5c134 in term 1.
I20250905 19:59:17.686857 16029 leader_election.cc:304] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37dcf68961894c1d9d0e691812799a37, f67dee448000427bacbf8414c3a5c134; no voters: 84d3dcd725c64c339b1f0dcc03e5a605
I20250905 19:59:17.687497 16204 raft_consensus.cc:2802] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:17.687775 16204 raft_consensus.cc:491] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 84d3dcd725c64c339b1f0dcc03e5a605)
I20250905 19:59:17.688071 16204 raft_consensus.cc:3058] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:17.692708 16204 raft_consensus.cc:513] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:17.694228 16204 leader_election.cc:290] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [CANDIDATE]: Term 2 election: Requested vote from peers 84d3dcd725c64c339b1f0dcc03e5a605 (127.14.175.129:39405), 37dcf68961894c1d9d0e691812799a37 (127.14.175.131:40723)
I20250905 19:59:17.695237 15989 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a05c6028354e418c6ee7256bef1491" candidate_uuid: "f67dee448000427bacbf8414c3a5c134" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84d3dcd725c64c339b1f0dcc03e5a605"
I20250905 19:59:17.695240 16138 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a05c6028354e418c6ee7256bef1491" candidate_uuid: "f67dee448000427bacbf8414c3a5c134" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37dcf68961894c1d9d0e691812799a37"
I20250905 19:59:17.695885 16138 raft_consensus.cc:3058] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:17.696332 16028 leader_election.cc:343] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 84d3dcd725c64c339b1f0dcc03e5a605 (127.14.175.129:39405): Illegal state: cannot vote while shut down
I20250905 19:59:17.700273 16138 raft_consensus.cc:2466] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f67dee448000427bacbf8414c3a5c134 in term 2.
I20250905 19:59:17.701089 16029 leader_election.cc:304] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37dcf68961894c1d9d0e691812799a37, f67dee448000427bacbf8414c3a5c134; no voters: 84d3dcd725c64c339b1f0dcc03e5a605
I20250905 19:59:17.701685 16204 raft_consensus.cc:2802] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:17.702736 16204 raft_consensus.cc:695] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 2 LEADER]: Becoming Leader. State: Replica: f67dee448000427bacbf8414c3a5c134, State: Running, Role: LEADER
I20250905 19:59:17.703397 16204 consensus_queue.cc:237] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3dcd725c64c339b1f0dcc03e5a605" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 39405 } } peers { permanent_uuid: "f67dee448000427bacbf8414c3a5c134" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 35429 } } peers { permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 } }
I20250905 19:59:18.135627 16138 raft_consensus.cc:1273] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 2 FOLLOWER]: Refusing update from remote peer f67dee448000427bacbf8414c3a5c134: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:18.136899 16204 consensus_queue.cc:1035] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dcf68961894c1d9d0e691812799a37" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:18.146363 16028 consensus_queue.cc:579] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [LEADER]: The tablet replica hosted on peer 84d3dcd725c64c339b1f0dcc03e5a605 has failed
W20250905 19:59:18.146943 16028 consensus_peers.cc:489] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 -> Peer 84d3dcd725c64c339b1f0dcc03e5a605 (127.14.175.129:39405): Couldn't send request to peer 84d3dcd725c64c339b1f0dcc03e5a605. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:18.356799 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
I20250905 19:59:18.373055 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:18.391291 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 19:59:18.398836 15038 tablet_server.cc:178] TabletServer@127.14.175.130:0 shutting down...
I20250905 19:59:18.412880 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:18.413395 15038 tablet_replica.cc:331] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134: stopping tablet replica
I20250905 19:59:18.413839 15038 raft_consensus.cc:2241] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:59:18.414604 15038 raft_consensus.cc:2270] T 78a05c6028354e418c6ee7256bef1491 P f67dee448000427bacbf8414c3a5c134 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:18.432883 15038 tablet_server.cc:195] TabletServer@127.14.175.130:0 shutdown complete.
I20250905 19:59:18.440371 15038 tablet_server.cc:178] TabletServer@127.14.175.131:0 shutting down...
I20250905 19:59:18.454608 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:18.455202 15038 tablet_replica.cc:331] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37: stopping tablet replica
I20250905 19:59:18.455693 15038 raft_consensus.cc:2241] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:18.456061 15038 raft_consensus.cc:2270] T 78a05c6028354e418c6ee7256bef1491 P 37dcf68961894c1d9d0e691812799a37 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:18.475996 15038 tablet_server.cc:195] TabletServer@127.14.175.131:0 shutdown complete.
I20250905 19:59:18.483395 15038 master.cc:561] Master@127.14.175.190:33213 shutting down...
I20250905 19:59:18.499300 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:18.499770 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:18.500029 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P c20fa7fd2841421695928da3e69f177b: stopping tablet replica
I20250905 19:59:18.516696 15038 master.cc:583] Master@127.14.175.190:33213 shutdown complete.
[ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (4065 ms)
[ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20250905 19:59:18.545991 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:44901
I20250905 19:59:18.547230 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:18.551879 16219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:18.553166 16220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:18.554900 16222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:18.555557 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:18.556288 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:18.556456 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:18.556599 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102358556580 us; error 0 us; skew 500 ppm
I20250905 19:59:18.557119 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:18.559309 15038 webserver.cc:480] Webserver started at http://127.14.175.190:38547/ using document root <none> and password file <none>
I20250905 19:59:18.559769 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:18.559940 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:18.560187 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:18.561193 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "fa9de6704aef40b6b5e62f92e42d6409"
format_stamp: "Formatted at 2025-09-05 19:59:18 on dist-test-slave-22s0"
I20250905 19:59:18.565424 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 19:59:18.568661 16227 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:18.569370 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:59:18.569636 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "fa9de6704aef40b6b5e62f92e42d6409"
format_stamp: "Formatted at 2025-09-05 19:59:18 on dist-test-slave-22s0"
I20250905 19:59:18.569891 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:18.579370 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:18.580413 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:18.616775 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:44901
I20250905 19:59:18.616875 16278 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:44901 every 8 connection(s)
I20250905 19:59:18.620510 16279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:18.630082 16279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409: Bootstrap starting.
I20250905 19:59:18.634600 16279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:18.638615 16279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409: No bootstrap required, opened a new log
I20250905 19:59:18.640538 16279 raft_consensus.cc:357] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9de6704aef40b6b5e62f92e42d6409" member_type: VOTER }
I20250905 19:59:18.640918 16279 raft_consensus.cc:383] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:18.641106 16279 raft_consensus.cc:738] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9de6704aef40b6b5e62f92e42d6409, State: Initialized, Role: FOLLOWER
I20250905 19:59:18.641599 16279 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9de6704aef40b6b5e62f92e42d6409" member_type: VOTER }
I20250905 19:59:18.642022 16279 raft_consensus.cc:397] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:18.642212 16279 raft_consensus.cc:491] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:18.642416 16279 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:18.646806 16279 raft_consensus.cc:513] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9de6704aef40b6b5e62f92e42d6409" member_type: VOTER }
I20250905 19:59:18.647375 16279 leader_election.cc:304] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa9de6704aef40b6b5e62f92e42d6409; no voters:
I20250905 19:59:18.648481 16279 leader_election.cc:290] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:18.648811 16282 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:18.650149 16282 raft_consensus.cc:695] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [term 1 LEADER]: Becoming Leader. State: Replica: fa9de6704aef40b6b5e62f92e42d6409, State: Running, Role: LEADER
I20250905 19:59:18.650873 16282 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9de6704aef40b6b5e62f92e42d6409" member_type: VOTER }
I20250905 19:59:18.651360 16279 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:18.653535 16283 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa9de6704aef40b6b5e62f92e42d6409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9de6704aef40b6b5e62f92e42d6409" member_type: VOTER } }
I20250905 19:59:18.654193 16283 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:18.653543 16284 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa9de6704aef40b6b5e62f92e42d6409. Latest consensus state: current_term: 1 leader_uuid: "fa9de6704aef40b6b5e62f92e42d6409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9de6704aef40b6b5e62f92e42d6409" member_type: VOTER } }
I20250905 19:59:18.655386 16284 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:18.657408 16288 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:18.661927 16288 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:18.665663 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:59:18.670065 16288 catalog_manager.cc:1349] Generated new cluster ID: 240503b4e4074d4090c526c31a084078
I20250905 19:59:18.670327 16288 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:18.683256 16288 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:59:18.684458 16288 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:18.697923 16288 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409: Generated new TSK 0
I20250905 19:59:18.698555 16288 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:18.731949 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:18.737511 16300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:18.738818 16301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:18.740345 16303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:18.740856 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:18.741551 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:18.741729 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:18.741858 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102358741846 us; error 0 us; skew 500 ppm
I20250905 19:59:18.742305 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:18.744647 15038 webserver.cc:480] Webserver started at http://127.14.175.129:37175/ using document root <none> and password file <none>
I20250905 19:59:18.745095 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:18.745245 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:18.745452 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:18.746464 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "8af247b39fc24c49bcfa24ac8ddecfe4"
format_stamp: "Formatted at 2025-09-05 19:59:18 on dist-test-slave-22s0"
I20250905 19:59:18.750948 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250905 19:59:18.754086 16308 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:18.754805 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 19:59:18.755076 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "8af247b39fc24c49bcfa24ac8ddecfe4"
format_stamp: "Formatted at 2025-09-05 19:59:18 on dist-test-slave-22s0"
I20250905 19:59:18.755332 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:18.773517 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:18.774685 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:18.776075 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:18.778162 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:18.778326 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:18.778609 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:18.778753 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:18.816354 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:45193
I20250905 19:59:18.816429 16370 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:45193 every 8 connection(s)
I20250905 19:59:18.829970 16371 heartbeater.cc:344] Connected to a master server at 127.14.175.190:44901
I20250905 19:59:18.830288 16371 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:18.830964 16371 heartbeater.cc:507] Master 127.14.175.190:44901 requested a full tablet report, sending...
I20250905 19:59:18.832737 16244 ts_manager.cc:194] Registered new tserver with Master: 8af247b39fc24c49bcfa24ac8ddecfe4 (127.14.175.129:45193)
I20250905 19:59:18.832857 15038 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.01335703s
I20250905 19:59:18.834722 16244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36968
I20250905 19:59:18.852000 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
I20250905 19:59:18.865316 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:18.880429 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 19:59:18.888866 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:59:18.891597 15038 tablet_server.cc:93] deduplicated master addresses: 127.14.175.190:44901
W20250905 19:59:18.894652 16381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:18.895972 16382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:18.896636 16384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:18.896831 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:18.897912 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:18.898211 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:18.898412 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102358898393 us; error 0 us; skew 500 ppm
I20250905 19:59:18.898960 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:18.901075 15038 webserver.cc:480] Webserver started at http://127.14.175.129:37175/ using document root <none> and password file <none>
I20250905 19:59:18.901512 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:18.901662 15038 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250905 19:59:18.905221 15038 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250905 19:59:18.907799 16389 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:18.908478 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250905 19:59:18.908739 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "8af247b39fc24c49bcfa24ac8ddecfe4"
format_stamp: "Formatted at 2025-09-05 19:59:18 on dist-test-slave-22s0"
I20250905 19:59:18.909021 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:18.920846 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:18.921805 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:18.923226 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:18.925383 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:18.925570 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:18.925803 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:18.925951 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:18.962622 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:45193
I20250905 19:59:18.962721 16451 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:45193 every 8 connection(s)
I20250905 19:59:18.969326 15038 tablet_server.cc:178] TabletServer@127.14.175.129:45193 shutting down...
I20250905 19:59:18.983762 16452 heartbeater.cc:344] Connected to a master server at 127.14.175.190:44901
I20250905 19:59:18.984170 16452 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:18.985200 16452 heartbeater.cc:507] Master 127.14.175.190:44901 requested a full tablet report, sending...
I20250905 19:59:18.987890 16244 ts_manager.cc:194] Re-registered known tserver with Master: 8af247b39fc24c49bcfa24ac8ddecfe4 (127.14.175.129:45193)
I20250905 19:59:18.989639 16244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36986
I20250905 19:59:18.992731 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:19.007897 15038 tablet_server.cc:195] TabletServer@127.14.175.129:45193 shutdown complete.
I20250905 19:59:19.014925 15038 master.cc:561] Master@127.14.175.190:44901 shutting down...
I20250905 19:59:19.027900 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:19.028407 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:19.028797 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P fa9de6704aef40b6b5e62f92e42d6409: stopping tablet replica
I20250905 19:59:19.037506 15038 master.cc:583] Master@127.14.175.190:44901 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (512 ms)
[ RUN ] TsTabletManagerITest.TestTableStats
I20250905 19:59:19.058744 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:38383,127.14.175.189:35421,127.14.175.188:40999
I20250905 19:59:19.060046 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:19.064610 16459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:19.065586 16460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:19.067296 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:19.067255 16462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:19.068209 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:19.068387 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:19.068503 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102359068493 us; error 0 us; skew 500 ppm
I20250905 19:59:19.068960 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:19.071125 15038 webserver.cc:480] Webserver started at http://127.14.175.190:46611/ using document root <none> and password file <none>
I20250905 19:59:19.071554 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:19.071702 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:19.071910 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:19.072926 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "865bfcc36a0b4876951dd6449b79ba9d"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:19.077072 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 19:59:19.080123 16467 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:19.080780 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 19:59:19.081005 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "865bfcc36a0b4876951dd6449b79ba9d"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:19.081235 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:19.094863 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:19.095798 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:19.130188 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:38383
I20250905 19:59:19.130291 16518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:38383 every 8 connection(s)
I20250905 19:59:19.133857 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:59:19.133935 16519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:59:19.139212 16521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:19.139436 16522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:19.141487 16519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } has no permanent_uuid. Determining permanent_uuid...
W20250905 19:59:19.144107 16524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:19.144726 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:19.145607 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:19.145843 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:19.146013 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102359145995 us; error 0 us; skew 500 ppm
I20250905 19:59:19.146536 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:19.149366 15038 webserver.cc:480] Webserver started at http://127.14.175.189:39111/ using document root <none> and password file <none>
I20250905 19:59:19.149837 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:19.150010 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:19.150239 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:19.151469 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-1-root/instance:
uuid: "5c2a729c06ee40b39008df4cc397d7d3"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:19.156641 16519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:59:19.156939 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
W20250905 19:59:19.159139 16471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.175.189:35421: connect: Connection refused (error 111)
I20250905 19:59:19.161305 16532 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:19.162089 15038 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 19:59:19.162367 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-1-root
uuid: "5c2a729c06ee40b39008df4cc397d7d3"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
W20250905 19:59:19.162806 16519 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.14.175.189:35421: Network error: Client connection negotiation failed: client connection to 127.14.175.189:35421: connect: Connection refused (error 111)
I20250905 19:59:19.162671 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:19.171247 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:19.172202 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:19.201294 16519 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } attempt: 1
W20250905 19:59:19.206636 16519 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.14.175.189:35421: Network error: Client connection negotiation failed: client connection to 127.14.175.189:35421: connect: Connection refused (error 111)
I20250905 19:59:19.208186 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.189:35421
I20250905 19:59:19.208271 16583 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.189:35421 every 8 connection(s)
I20250905 19:59:19.211716 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:59:19.211874 16584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:59:19.216802 16586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:19.219050 16587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:19.220763 16589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:19.221052 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:19.220916 16584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:59:19.221855 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:19.222061 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:19.222268 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102359222249 us; error 0 us; skew 500 ppm
I20250905 19:59:19.223127 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:19.226054 15038 webserver.cc:480] Webserver started at http://127.14.175.188:36779/ using document root <none> and password file <none>
I20250905 19:59:19.226692 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:19.226929 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:19.227222 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:19.228332 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-2-root/instance:
uuid: "13d4d4a7aaf346b89df6aba94104e180"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:19.232466 16584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:59:19.233219 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 19:59:19.237514 16596 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:19.238313 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250905 19:59:19.238651 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-2-root
uuid: "13d4d4a7aaf346b89df6aba94104e180"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:19.238991 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:19.244428 16584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:59:19.247124 16519 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } attempt: 2
W20250905 19:59:19.249716 16584 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.14.175.188:40999: Network error: Client connection negotiation failed: client connection to 127.14.175.188:40999: connect: Connection refused (error 111)
I20250905 19:59:19.256585 16519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } has no permanent_uuid. Determining permanent_uuid...
W20250905 19:59:19.260428 16519 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.14.175.188:40999: Network error: Client connection negotiation failed: client connection to 127.14.175.188:40999: connect: Connection refused (error 111)
I20250905 19:59:19.261178 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:19.262143 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:19.285177 16584 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } attempt: 1
W20250905 19:59:19.289635 16584 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.14.175.188:40999: Network error: Client connection negotiation failed: client connection to 127.14.175.188:40999: connect: Connection refused (error 111)
I20250905 19:59:19.293970 16519 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } attempt: 1
I20250905 19:59:19.297845 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.188:40999
I20250905 19:59:19.297971 16648 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.188:40999 every 8 connection(s)
I20250905 19:59:19.301690 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:59:19.302484 16650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:19.309852 16650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:59:19.315110 16519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: Bootstrap starting.
I20250905 19:59:19.320417 16519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:19.320436 16650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:59:19.324885 16519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: No bootstrap required, opened a new log
I20250905 19:59:19.326938 16519 raft_consensus.cc:357] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:19.327494 16519 raft_consensus.cc:383] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:19.327829 16519 raft_consensus.cc:738] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865bfcc36a0b4876951dd6449b79ba9d, State: Initialized, Role: FOLLOWER
I20250905 19:59:19.328548 16519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:19.330433 16654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:19.331288 16654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:19.332080 16519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:19.332710 16650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:59:19.344313 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20250905 19:59:19.347230 16665 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 19:59:19.347527 16665 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 19:59:19.348169 16650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180: Bootstrap starting.
I20250905 19:59:19.352295 16650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:19.355883 16650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180: No bootstrap required, opened a new log
I20250905 19:59:19.357703 16650 raft_consensus.cc:357] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:19.358206 16650 raft_consensus.cc:383] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:19.358387 16650 raft_consensus.cc:738] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d4d4a7aaf346b89df6aba94104e180, State: Initialized, Role: FOLLOWER
I20250905 19:59:19.358922 16650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:19.360409 16667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:19.360946 16667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:19.361539 16650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:19.365217 16584 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } attempt: 2
W20250905 19:59:19.377102 16678 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 19:59:19.377384 16678 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 19:59:19.380614 16584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: Bootstrap starting.
I20250905 19:59:19.384738 16584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:19.388299 16584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: No bootstrap required, opened a new log
I20250905 19:59:19.390076 16584 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:19.390592 16584 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:19.390821 16584 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c2a729c06ee40b39008df4cc397d7d3, State: Initialized, Role: FOLLOWER
I20250905 19:59:19.391306 16584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:19.392810 16680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:19.393347 16680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:19.393923 16584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:19.394176 16667 raft_consensus.cc:491] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:19.394551 16667 raft_consensus.cc:513] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:19.396076 16667 leader_election.cc:290] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383), 5c2a729c06ee40b39008df4cc397d7d3 (127.14.175.189:35421)
I20250905 19:59:19.397258 16559 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13d4d4a7aaf346b89df6aba94104e180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c2a729c06ee40b39008df4cc397d7d3" is_pre_election: true
I20250905 19:59:19.397289 16494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13d4d4a7aaf346b89df6aba94104e180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865bfcc36a0b4876951dd6449b79ba9d" is_pre_election: true
I20250905 19:59:19.398000 16559 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d4d4a7aaf346b89df6aba94104e180 in term 0.
I20250905 19:59:19.398249 16494 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d4d4a7aaf346b89df6aba94104e180 in term 0.
I20250905 19:59:19.399116 16600 leader_election.cc:304] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d4d4a7aaf346b89df6aba94104e180, 5c2a729c06ee40b39008df4cc397d7d3; no voters:
I20250905 19:59:19.400025 16667 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:59:19.400454 16667 raft_consensus.cc:491] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:19.400792 16667 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:19.407089 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250905 19:59:19.408382 16667 raft_consensus.cc:513] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:19.408958 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:59:19.409804 16667 leader_election.cc:290] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 1 election: Requested vote from peers 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383), 5c2a729c06ee40b39008df4cc397d7d3 (127.14.175.189:35421)
I20250905 19:59:19.410879 16494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13d4d4a7aaf346b89df6aba94104e180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865bfcc36a0b4876951dd6449b79ba9d"
I20250905 19:59:19.411303 16494 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:19.411203 16559 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13d4d4a7aaf346b89df6aba94104e180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c2a729c06ee40b39008df4cc397d7d3"
I20250905 19:59:19.411779 16559 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:59:19.416510 16692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:19.416554 16494 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d4d4a7aaf346b89df6aba94104e180 in term 1.
I20250905 19:59:19.417802 16600 leader_election.cc:304] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d4d4a7aaf346b89df6aba94104e180, 865bfcc36a0b4876951dd6449b79ba9d; no voters:
I20250905 19:59:19.418412 16667 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 1 FOLLOWER]: Leader election won for term 1
W20250905 19:59:19.418874 16691 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 19:59:19.419229 16691 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 19:59:19.420051 16667 raft_consensus.cc:695] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 1 LEADER]: Becoming Leader. State: Replica: 13d4d4a7aaf346b89df6aba94104e180, State: Running, Role: LEADER
I20250905 19:59:19.420488 16559 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d4d4a7aaf346b89df6aba94104e180 in term 1.
I20250905 19:59:19.420941 16667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
W20250905 19:59:19.422880 16693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:19.425848 16696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:19.426539 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:19.426136 16694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13d4d4a7aaf346b89df6aba94104e180. Latest consensus state: current_term: 1 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:19.427124 16694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:19.427604 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:19.427958 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:19.428166 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102359428147 us; error 0 us; skew 500 ppm
I20250905 19:59:19.428339 16699 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:19.428769 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:19.431262 15038 webserver.cc:480] Webserver started at http://127.14.175.129:46759/ using document root <none> and password file <none>
I20250905 19:59:19.431710 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:19.431885 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:19.432119 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:19.433169 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "3a3f9980cc9e40efaf4855a979c04df2"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:19.433252 16699 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:19.437925 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 19:59:19.440891 16494 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 1 FOLLOWER]: Refusing update from remote peer 13d4d4a7aaf346b89df6aba94104e180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:19.440939 16559 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 1 FOLLOWER]: Refusing update from remote peer 13d4d4a7aaf346b89df6aba94104e180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:19.442371 16667 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:59:19.442960 16704 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:19.443154 16694 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:59:19.443804 15038 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250905 19:59:19.444092 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "3a3f9980cc9e40efaf4855a979c04df2"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:19.444438 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:19.455199 16680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13d4d4a7aaf346b89df6aba94104e180. Latest consensus state: current_term: 1 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:19.455842 16680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:19.459519 16654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13d4d4a7aaf346b89df6aba94104e180. Latest consensus state: current_term: 1 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:19.460175 16702 mvcc.cc:204] Tried to move back new op lower bound from 7197091264257912832 to 7197091264198176768. Current Snapshot: MvccSnapshot[applied={T|T < 7197091264257912832}]
I20250905 19:59:19.460196 16654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:19.462088 16667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:19.463049 16667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:19.463943 16699 catalog_manager.cc:1349] Generated new cluster ID: 84986f46af5743b49ebce666260696a5
I20250905 19:59:19.464231 16699 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:19.465999 16680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:19.466711 16680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:19.467403 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:19.468827 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:19.470546 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:19.471871 16694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:19.472515 16694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:19.473810 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:19.474167 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250905 19:59:19.474534 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:19.474823 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:19.475294 16654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:19.476166 16654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:19.503221 16699 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:59:19.504434 16699 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:19.536366 16699 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180: Generated new TSK 0
I20250905 19:59:19.537284 16699 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:19.551450 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:33875
I20250905 19:59:19.551530 16775 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:33875 every 8 connection(s)
I20250905 19:59:19.560209 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:59:19.573930 16776 heartbeater.cc:344] Connected to a master server at 127.14.175.188:40999
I20250905 19:59:19.574442 16776 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:19.575582 16776 heartbeater.cc:507] Master 127.14.175.188:40999 requested a full tablet report, sending...
W20250905 19:59:19.576335 16783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:19.578227 16614 ts_manager.cc:194] Registered new tserver with Master: 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875)
I20250905 19:59:19.578704 16777 heartbeater.cc:344] Connected to a master server at 127.14.175.189:35421
I20250905 19:59:19.579097 16777 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:19.579928 16777 heartbeater.cc:507] Master 127.14.175.189:35421 requested a full tablet report, sending...
I20250905 19:59:19.581013 16614 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52852
I20250905 19:59:19.582932 16549 ts_manager.cc:194] Registered new tserver with Master: 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875)
W20250905 19:59:19.589102 16784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:19.589784 16778 heartbeater.cc:344] Connected to a master server at 127.14.175.190:38383
I20250905 19:59:19.590258 16778 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:19.590978 16778 heartbeater.cc:507] Master 127.14.175.190:38383 requested a full tablet report, sending...
W20250905 19:59:19.591615 16786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:19.591662 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:19.592816 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:19.593036 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:19.593211 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102359593193 us; error 0 us; skew 500 ppm
I20250905 19:59:19.593421 16484 ts_manager.cc:194] Registered new tserver with Master: 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875)
I20250905 19:59:19.593732 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:19.596060 15038 webserver.cc:480] Webserver started at http://127.14.175.130:33969/ using document root <none> and password file <none>
I20250905 19:59:19.596508 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:19.596683 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:19.596916 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:19.597913 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-1-root/instance:
uuid: "df4471c1f39a4a5e823695f3794cc709"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:19.602186 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 19:59:19.605394 16791 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:19.606179 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:59:19.606453 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-1-root
uuid: "df4471c1f39a4a5e823695f3794cc709"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:19.606828 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:19.632923 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:19.634057 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:19.635967 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:19.638840 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:19.639074 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:19.639295 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:19.639451 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:19.684294 16776 heartbeater.cc:499] Master 127.14.175.188:40999 was elected leader, sending a full tablet report...
I20250905 19:59:19.715600 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.130:34159
I20250905 19:59:19.715700 16853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.130:34159 every 8 connection(s)
I20250905 19:59:19.736243 16854 heartbeater.cc:344] Connected to a master server at 127.14.175.188:40999
I20250905 19:59:19.736711 16854 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:19.737659 16854 heartbeater.cc:507] Master 127.14.175.188:40999 requested a full tablet report, sending...
I20250905 19:59:19.740134 16614 ts_manager.cc:194] Registered new tserver with Master: df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159)
I20250905 19:59:19.741858 16614 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52856
I20250905 19:59:19.748180 16855 heartbeater.cc:344] Connected to a master server at 127.14.175.189:35421
I20250905 19:59:19.748675 16855 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:19.749476 16855 heartbeater.cc:507] Master 127.14.175.189:35421 requested a full tablet report, sending...
I20250905 19:59:19.750655 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:59:19.750949 16857 heartbeater.cc:344] Connected to a master server at 127.14.175.190:38383
I20250905 19:59:19.751351 16857 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:19.751856 16549 ts_manager.cc:194] Registered new tserver with Master: df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159)
I20250905 19:59:19.752008 16857 heartbeater.cc:507] Master 127.14.175.190:38383 requested a full tablet report, sending...
I20250905 19:59:19.754477 16484 ts_manager.cc:194] Registered new tserver with Master: df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159)
W20250905 19:59:19.758144 16861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:19.760555 16862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:19.763584 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:19.763542 16864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:19.764739 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:19.764954 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:19.765143 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102359765127 us; error 0 us; skew 500 ppm
I20250905 19:59:19.765636 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:19.767911 15038 webserver.cc:480] Webserver started at http://127.14.175.131:36187/ using document root <none> and password file <none>
I20250905 19:59:19.768368 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:19.768559 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:19.768800 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:19.769789 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-2-root/instance:
uuid: "b0aaa68ee9814737a4ad3ab35dbbecba"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:19.773864 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 19:59:19.776862 16869 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:19.777500 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:59:19.777761 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-2-root
uuid: "b0aaa68ee9814737a4ad3ab35dbbecba"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:19.778023 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:19.830803 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:19.831929 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:19.833411 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:19.835368 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:19.835567 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:19.835788 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:19.835956 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:19.845047 16854 heartbeater.cc:499] Master 127.14.175.188:40999 was elected leader, sending a full tablet report...
I20250905 19:59:19.911675 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.131:41469
I20250905 19:59:19.911774 16931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.131:41469 every 8 connection(s)
I20250905 19:59:19.938694 16933 heartbeater.cc:344] Connected to a master server at 127.14.175.189:35421
I20250905 19:59:19.939144 16933 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:19.940086 16933 heartbeater.cc:507] Master 127.14.175.189:35421 requested a full tablet report, sending...
I20250905 19:59:19.943087 16935 heartbeater.cc:344] Connected to a master server at 127.14.175.190:38383
I20250905 19:59:19.943529 16932 heartbeater.cc:344] Connected to a master server at 127.14.175.188:40999
I20250905 19:59:19.943137 16549 ts_manager.cc:194] Registered new tserver with Master: b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:19.944039 16935 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:19.944216 16932 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:19.944962 16935 heartbeater.cc:507] Master 127.14.175.190:38383 requested a full tablet report, sending...
I20250905 19:59:19.945017 16932 heartbeater.cc:507] Master 127.14.175.188:40999 requested a full tablet report, sending...
I20250905 19:59:19.947427 16614 ts_manager.cc:194] Registered new tserver with Master: b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:19.947845 16484 ts_manager.cc:194] Registered new tserver with Master: b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:19.948055 15038 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01419441s
I20250905 19:59:19.949126 16614 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52872
I20250905 19:59:19.984548 16614 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20250905 19:59:19.986652 16614 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:59:20.046584 16897 tablet_service.cc:1468] Processing CreateTablet for tablet 09dda0a688e441c0afdf52ec0e2a903c (DEFAULT_TABLE table=test-table [id=2518142617a742a6a53c69110c8d664e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:20.047488 16896 tablet_service.cc:1468] Processing CreateTablet for tablet 6183f63c34da4aa6a03ca53fff820702 (DEFAULT_TABLE table=test-table [id=2518142617a742a6a53c69110c8d664e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:20.048161 16897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09dda0a688e441c0afdf52ec0e2a903c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:20.048923 16896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6183f63c34da4aa6a03ca53fff820702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:20.052017 16932 heartbeater.cc:499] Master 127.14.175.188:40999 was elected leader, sending a full tablet report...
I20250905 19:59:20.058127 16819 tablet_service.cc:1468] Processing CreateTablet for tablet 09dda0a688e441c0afdf52ec0e2a903c (DEFAULT_TABLE table=test-table [id=2518142617a742a6a53c69110c8d664e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:20.059681 16819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09dda0a688e441c0afdf52ec0e2a903c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:20.063691 16895 tablet_service.cc:1468] Processing CreateTablet for tablet a9cf7a45fc234e7db1286cbf5f8f0f0b (DEFAULT_TABLE table=test-table [id=2518142617a742a6a53c69110c8d664e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:20.065186 16895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cf7a45fc234e7db1286cbf5f8f0f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:20.070003 16818 tablet_service.cc:1468] Processing CreateTablet for tablet 6183f63c34da4aa6a03ca53fff820702 (DEFAULT_TABLE table=test-table [id=2518142617a742a6a53c69110c8d664e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:20.070742 16817 tablet_service.cc:1468] Processing CreateTablet for tablet a9cf7a45fc234e7db1286cbf5f8f0f0b (DEFAULT_TABLE table=test-table [id=2518142617a742a6a53c69110c8d664e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:20.071507 16818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6183f63c34da4aa6a03ca53fff820702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:20.072271 16817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cf7a45fc234e7db1286cbf5f8f0f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:20.073606 16953 tablet_bootstrap.cc:492] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba: Bootstrap starting.
I20250905 19:59:20.082901 16737 tablet_service.cc:1468] Processing CreateTablet for tablet a9cf7a45fc234e7db1286cbf5f8f0f0b (DEFAULT_TABLE table=test-table [id=2518142617a742a6a53c69110c8d664e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:20.080374 16741 tablet_service.cc:1468] Processing CreateTablet for tablet 09dda0a688e441c0afdf52ec0e2a903c (DEFAULT_TABLE table=test-table [id=2518142617a742a6a53c69110c8d664e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:20.085124 16737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cf7a45fc234e7db1286cbf5f8f0f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:20.081260 16739 tablet_service.cc:1468] Processing CreateTablet for tablet 6183f63c34da4aa6a03ca53fff820702 (DEFAULT_TABLE table=test-table [id=2518142617a742a6a53c69110c8d664e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:20.086640 16741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09dda0a688e441c0afdf52ec0e2a903c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:20.089076 16739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6183f63c34da4aa6a03ca53fff820702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:20.089604 16953 tablet_bootstrap.cc:654] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:20.119411 16955 tablet_bootstrap.cc:492] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709: Bootstrap starting.
I20250905 19:59:20.120968 16953 tablet_bootstrap.cc:492] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba: No bootstrap required, opened a new log
I20250905 19:59:20.121623 16953 ts_tablet_manager.cc:1397] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba: Time spent bootstrapping tablet: real 0.048s user 0.017s sys 0.006s
I20250905 19:59:20.124574 16956 tablet_bootstrap.cc:492] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2: Bootstrap starting.
I20250905 19:59:20.126284 16953 raft_consensus.cc:357] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:20.127024 16953 raft_consensus.cc:383] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:20.127321 16953 raft_consensus.cc:738] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0aaa68ee9814737a4ad3ab35dbbecba, State: Initialized, Role: FOLLOWER
I20250905 19:59:20.128088 16953 consensus_queue.cc:260] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:20.135509 16955 tablet_bootstrap.cc:654] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:20.146198 16953 ts_tablet_manager.cc:1428] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba: Time spent starting tablet: real 0.024s user 0.004s sys 0.007s
I20250905 19:59:20.150039 16958 raft_consensus.cc:491] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:20.150599 16958 raft_consensus.cc:513] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:20.154824 16958 leader_election.cc:290] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159), 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875)
I20250905 19:59:20.156272 16955 tablet_bootstrap.cc:492] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709: No bootstrap required, opened a new log
I20250905 19:59:20.156849 16955 ts_tablet_manager.cc:1397] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.013s
I20250905 19:59:20.160966 16955 raft_consensus.cc:357] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:20.161736 16955 raft_consensus.cc:383] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:20.162056 16955 raft_consensus.cc:738] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4471c1f39a4a5e823695f3794cc709, State: Initialized, Role: FOLLOWER
I20250905 19:59:20.162806 16955 consensus_queue.cc:260] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:20.164403 16953 tablet_bootstrap.cc:492] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba: Bootstrap starting.
I20250905 19:59:20.173776 16953 tablet_bootstrap.cc:654] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:20.182591 16751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c" candidate_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3f9980cc9e40efaf4855a979c04df2" is_pre_election: true
W20250905 19:59:20.184911 16871 leader_election.cc:343] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:59:20.186617 16829 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c" candidate_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709" is_pre_election: true
I20250905 19:59:20.187587 16829 raft_consensus.cc:2466] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0aaa68ee9814737a4ad3ab35dbbecba in term 0.
I20250905 19:59:20.189411 16870 leader_election.cc:304] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709; no voters: 3a3f9980cc9e40efaf4855a979c04df2
I20250905 19:59:20.191692 16958 raft_consensus.cc:2802] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:59:20.192180 16958 raft_consensus.cc:491] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:20.192602 16958 raft_consensus.cc:3058] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:20.200775 16956 tablet_bootstrap.cc:654] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:20.212085 16955 ts_tablet_manager.cc:1428] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709: Time spent starting tablet: real 0.055s user 0.012s sys 0.040s
I20250905 19:59:20.212182 16958 raft_consensus.cc:513] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:20.217058 16955 tablet_bootstrap.cc:492] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709: Bootstrap starting.
I20250905 19:59:20.218660 16956 tablet_bootstrap.cc:492] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2: No bootstrap required, opened a new log
I20250905 19:59:20.218601 16953 tablet_bootstrap.cc:492] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba: No bootstrap required, opened a new log
I20250905 19:59:20.219296 16956 ts_tablet_manager.cc:1397] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.007s
I20250905 19:59:20.219372 16953 ts_tablet_manager.cc:1397] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba: Time spent bootstrapping tablet: real 0.055s user 0.022s sys 0.019s
I20250905 19:59:20.222652 16953 raft_consensus.cc:357] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.223501 16953 raft_consensus.cc:383] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:20.223600 16829 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c" candidate_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709"
I20250905 19:59:20.223935 16953 raft_consensus.cc:738] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0aaa68ee9814737a4ad3ab35dbbecba, State: Initialized, Role: FOLLOWER
I20250905 19:59:20.223955 16955 tablet_bootstrap.cc:654] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:20.224190 16829 raft_consensus.cc:3058] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:20.224968 16953 consensus_queue.cc:260] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.232540 16953 ts_tablet_manager.cc:1428] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250905 19:59:20.233654 16953 tablet_bootstrap.cc:492] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba: Bootstrap starting.
I20250905 19:59:20.246140 16958 leader_election.cc:290] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 1 election: Requested vote from peers df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159), 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875)
I20250905 19:59:20.254637 16956 raft_consensus.cc:357] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:20.256392 16751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c" candidate_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3f9980cc9e40efaf4855a979c04df2"
I20250905 19:59:20.256680 16956 raft_consensus.cc:383] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:20.252508 16829 raft_consensus.cc:2466] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0aaa68ee9814737a4ad3ab35dbbecba in term 1.
I20250905 19:59:20.258898 16956 raft_consensus.cc:738] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3f9980cc9e40efaf4855a979c04df2, State: Initialized, Role: FOLLOWER
I20250905 19:59:20.260067 16956 consensus_queue.cc:260] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:20.261154 16870 leader_election.cc:304] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709; no voters:
I20250905 19:59:20.261806 16953 tablet_bootstrap.cc:654] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:20.263690 16751 raft_consensus.cc:3058] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:20.262264 16955 tablet_bootstrap.cc:492] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709: No bootstrap required, opened a new log
I20250905 19:59:20.263927 16958 raft_consensus.cc:2802] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:20.264341 16955 ts_tablet_manager.cc:1397] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709: Time spent bootstrapping tablet: real 0.048s user 0.024s sys 0.019s
I20250905 19:59:20.266396 16958 raft_consensus.cc:695] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 LEADER]: Becoming Leader. State: Replica: b0aaa68ee9814737a4ad3ab35dbbecba, State: Running, Role: LEADER
I20250905 19:59:20.267501 16958 consensus_queue.cc:237] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:20.267858 16955 raft_consensus.cc:357] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.268597 16955 raft_consensus.cc:383] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:20.268906 16955 raft_consensus.cc:738] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4471c1f39a4a5e823695f3794cc709, State: Initialized, Role: FOLLOWER
I20250905 19:59:20.269810 16955 consensus_queue.cc:260] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.276811 16955 ts_tablet_manager.cc:1428] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250905 19:59:20.277869 16955 tablet_bootstrap.cc:492] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709: Bootstrap starting.
I20250905 19:59:20.281780 16957 raft_consensus.cc:491] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:20.282322 16957 raft_consensus.cc:513] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.284344 16955 tablet_bootstrap.cc:654] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:20.284483 16957 leader_election.cc:290] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875), df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159)
I20250905 19:59:20.286584 16829 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709" is_pre_election: true
I20250905 19:59:20.287478 16829 raft_consensus.cc:2466] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0aaa68ee9814737a4ad3ab35dbbecba in term 0.
I20250905 19:59:20.288720 16870 leader_election.cc:304] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709; no voters:
I20250905 19:59:20.290236 16750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3f9980cc9e40efaf4855a979c04df2" is_pre_election: true
I20250905 19:59:20.292605 16957 raft_consensus.cc:2802] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:59:20.293151 16957 raft_consensus.cc:491] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:20.293596 16957 raft_consensus.cc:3058] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:20.294939 16751 raft_consensus.cc:2466] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0aaa68ee9814737a4ad3ab35dbbecba in term 1.
I20250905 19:59:20.296155 16955 tablet_bootstrap.cc:492] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709: No bootstrap required, opened a new log
I20250905 19:59:20.296720 16955 ts_tablet_manager.cc:1397] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
W20250905 19:59:20.298842 16871 leader_election.cc:343] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:59:20.299867 16955 raft_consensus.cc:357] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.300751 16955 raft_consensus.cc:383] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:20.301141 16955 raft_consensus.cc:738] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4471c1f39a4a5e823695f3794cc709, State: Initialized, Role: FOLLOWER
I20250905 19:59:20.302413 16955 consensus_queue.cc:260] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.307829 16955 ts_tablet_manager.cc:1428] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20250905 19:59:20.315213 16960 raft_consensus.cc:491] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:20.315765 16960 raft_consensus.cc:513] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.322218 16960 leader_election.cc:290] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875), b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:20.323343 16956 ts_tablet_manager.cc:1428] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2: Time spent starting tablet: real 0.104s user 0.010s sys 0.005s
I20250905 19:59:20.326797 16956 tablet_bootstrap.cc:492] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2: Bootstrap starting.
I20250905 19:59:20.333684 16956 tablet_bootstrap.cc:654] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:20.323932 16957 raft_consensus.cc:513] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.346659 16953 tablet_bootstrap.cc:492] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba: No bootstrap required, opened a new log
I20250905 19:59:20.347213 16953 ts_tablet_manager.cc:1397] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba: Time spent bootstrapping tablet: real 0.114s user 0.026s sys 0.026s
I20250905 19:59:20.347433 16751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3f9980cc9e40efaf4855a979c04df2"
I20250905 19:59:20.348978 16829 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709"
I20250905 19:59:20.348289 16614 catalog_manager.cc:5582] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba reported cstate change: term changed from 0 to 1, leader changed from <none> to b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131). New cstate: current_term: 1 leader_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:20.349594 16829 raft_consensus.cc:3058] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:20.353235 16665 catalog_manager.cc:1261] Loaded cluster ID: 84986f46af5743b49ebce666260696a5
I20250905 19:59:20.353554 16665 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: loading cluster ID for follower catalog manager: success
I20250905 19:59:20.357604 16829 raft_consensus.cc:2466] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0aaa68ee9814737a4ad3ab35dbbecba in term 1.
I20250905 19:59:20.360357 16870 leader_election.cc:304] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709; no voters:
I20250905 19:59:20.360707 16665 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: acquiring CA information for follower catalog manager: success
I20250905 19:59:20.361285 16958 raft_consensus.cc:2802] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 FOLLOWER]: Leader election won for term 1
W20250905 19:59:20.365621 16871 leader_election.cc:343] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:59:20.365963 16953 raft_consensus.cc:357] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.366842 16953 raft_consensus.cc:383] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:20.367210 16953 raft_consensus.cc:738] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0aaa68ee9814737a4ad3ab35dbbecba, State: Initialized, Role: FOLLOWER
I20250905 19:59:20.367466 16957 leader_election.cc:290] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 1 election: Requested vote from peers 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875), df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159)
I20250905 19:59:20.367918 16953 consensus_queue.cc:260] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.370479 16953 ts_tablet_manager.cc:1428] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20250905 19:59:20.371718 16957 consensus_queue.cc:1035] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:59:20.375535 16665 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 19:59:20.377936 16958 raft_consensus.cc:695] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 LEADER]: Becoming Leader. State: Replica: b0aaa68ee9814737a4ad3ab35dbbecba, State: Running, Role: LEADER
I20250905 19:59:20.378841 16958 consensus_queue.cc:237] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.379626 16751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3f9980cc9e40efaf4855a979c04df2" is_pre_election: true
W20250905 19:59:20.386489 16793 leader_election.cc:343] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:59:20.413625 16971 consensus_queue.cc:1035] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:59:20.424587 16691 catalog_manager.cc:1261] Loaded cluster ID: 84986f46af5743b49ebce666260696a5
I20250905 19:59:20.424945 16691 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: loading cluster ID for follower catalog manager: success
W20250905 19:59:20.428969 16871 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.175.129:33875, user_credentials={real_user=slave}} blocked reactor thread for 62597.1us
I20250905 19:59:20.431713 16691 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: acquiring CA information for follower catalog manager: success
I20250905 19:59:20.453521 16691 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 19:59:20.469717 16612 catalog_manager.cc:5582] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba reported cstate change: term changed from 0 to 1, leader changed from <none> to b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131). New cstate: current_term: 1 leader_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:20.472985 16956 tablet_bootstrap.cc:492] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2: No bootstrap required, opened a new log
I20250905 19:59:20.473521 16956 ts_tablet_manager.cc:1397] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2: Time spent bootstrapping tablet: real 0.147s user 0.040s sys 0.041s
I20250905 19:59:20.475713 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" is_pre_election: true
I20250905 19:59:20.476483 16907 raft_consensus.cc:2466] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df4471c1f39a4a5e823695f3794cc709 in term 0.
I20250905 19:59:20.476595 16956 raft_consensus.cc:357] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.477224 16956 raft_consensus.cc:383] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:20.477463 16956 raft_consensus.cc:738] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3f9980cc9e40efaf4855a979c04df2, State: Initialized, Role: FOLLOWER
I20250905 19:59:20.477492 16793 leader_election.cc:304] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709; no voters: 3a3f9980cc9e40efaf4855a979c04df2
I20250905 19:59:20.480435 16960 raft_consensus.cc:2802] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:59:20.480983 16960 raft_consensus.cc:491] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:20.480484 16956 consensus_queue.cc:260] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.481310 16960 raft_consensus.cc:3058] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:59:20.482534 16871 consensus_peers.cc:489] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba -> Peer 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875): Couldn't send request to peer 3a3f9980cc9e40efaf4855a979c04df2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:20.491091 16956 ts_tablet_manager.cc:1428] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2: Time spent starting tablet: real 0.017s user 0.008s sys 0.001s
I20250905 19:59:20.494285 16956 tablet_bootstrap.cc:492] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2: Bootstrap starting.
I20250905 19:59:20.500268 16956 tablet_bootstrap.cc:654] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:20.501926 16972 consensus_queue.cc:1035] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:59:20.507799 16960 raft_consensus.cc:513] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.510553 16751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3f9980cc9e40efaf4855a979c04df2"
W20250905 19:59:20.512059 16793 leader_election.cc:343] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:59:20.517452 16960 leader_election.cc:290] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875), b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:20.521166 16956 tablet_bootstrap.cc:492] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2: No bootstrap required, opened a new log
I20250905 19:59:20.521670 16956 ts_tablet_manager.cc:1397] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.005s
I20250905 19:59:20.524039 16956 raft_consensus.cc:357] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.524551 16956 raft_consensus.cc:383] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:20.524838 16956 raft_consensus.cc:738] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3f9980cc9e40efaf4855a979c04df2, State: Initialized, Role: FOLLOWER
I20250905 19:59:20.525463 16956 consensus_queue.cc:260] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.527206 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba"
I20250905 19:59:20.527542 16956 ts_tablet_manager.cc:1428] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250905 19:59:20.527796 16907 raft_consensus.cc:3058] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:20.534487 16907 raft_consensus.cc:2466] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df4471c1f39a4a5e823695f3794cc709 in term 1.
I20250905 19:59:20.535454 16793 leader_election.cc:304] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709; no voters: 3a3f9980cc9e40efaf4855a979c04df2
I20250905 19:59:20.536334 16960 raft_consensus.cc:2802] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:20.536836 16960 raft_consensus.cc:695] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 1 LEADER]: Becoming Leader. State: Replica: df4471c1f39a4a5e823695f3794cc709, State: Running, Role: LEADER
I20250905 19:59:20.537642 16960 consensus_queue.cc:237] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.539933 16967 raft_consensus.cc:491] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:20.540411 16967 raft_consensus.cc:513] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.542865 16967 leader_election.cc:290] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159), b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:20.568291 16751 raft_consensus.cc:3058] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:20.579594 16612 catalog_manager.cc:5582] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 reported cstate change: term changed from 0 to 1, leader changed from <none> to df4471c1f39a4a5e823695f3794cc709 (127.14.175.130). New cstate: current_term: 1 leader_uuid: "df4471c1f39a4a5e823695f3794cc709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:20.583467 16828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709" is_pre_election: true
I20250905 19:59:20.586834 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" is_pre_election: true
I20250905 19:59:20.589349 16710 leader_election.cc:304] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3f9980cc9e40efaf4855a979c04df2; no voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709
I20250905 19:59:20.591122 16967 raft_consensus.cc:2747] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:59:20.606856 16967 raft_consensus.cc:491] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:20.607273 16967 raft_consensus.cc:513] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:20.609068 16967 leader_election.cc:290] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159), b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:20.609638 16828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709" is_pre_election: true
I20250905 19:59:20.611276 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" is_pre_election: true
I20250905 19:59:20.611929 16907 raft_consensus.cc:2391] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a3f9980cc9e40efaf4855a979c04df2 in current term 1: Already voted for candidate df4471c1f39a4a5e823695f3794cc709 in this term.
I20250905 19:59:20.612685 16710 leader_election.cc:304] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3f9980cc9e40efaf4855a979c04df2; no voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709
I20250905 19:59:20.613273 16967 raft_consensus.cc:3058] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:20.618870 16967 raft_consensus.cc:2747] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:59:20.644524 16960 consensus_queue.cc:1035] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:59:21.084628 16988 raft_consensus.cc:491] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 13d4d4a7aaf346b89df6aba94104e180)
I20250905 19:59:21.085316 16988 raft_consensus.cc:513] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:21.088879 16988 leader_election.cc:290] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c2a729c06ee40b39008df4cc397d7d3 (127.14.175.189:35421), 13d4d4a7aaf346b89df6aba94104e180 (127.14.175.188:40999)
I20250905 19:59:21.109941 16624 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "865bfcc36a0b4876951dd6449b79ba9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "13d4d4a7aaf346b89df6aba94104e180" is_pre_election: true
I20250905 19:59:21.114223 16989 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 13d4d4a7aaf346b89df6aba94104e180)
I20250905 19:59:21.114805 16989 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:21.116845 16989 leader_election.cc:290] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383), 13d4d4a7aaf346b89df6aba94104e180 (127.14.175.188:40999)
I20250905 19:59:21.117610 16494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c2a729c06ee40b39008df4cc397d7d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "865bfcc36a0b4876951dd6449b79ba9d" is_pre_election: true
I20250905 19:59:21.118296 16494 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c2a729c06ee40b39008df4cc397d7d3 in term 1.
I20250905 19:59:21.119428 16536 leader_election.cc:304] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c2a729c06ee40b39008df4cc397d7d3, 865bfcc36a0b4876951dd6449b79ba9d; no voters:
I20250905 19:59:21.120167 16989 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:21.120539 16989 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 13d4d4a7aaf346b89df6aba94104e180)
I20250905 19:59:21.120945 16989 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:21.128233 16989 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:21.130787 16989 leader_election.cc:290] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [CANDIDATE]: Term 2 election: Requested vote from peers 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383), 13d4d4a7aaf346b89df6aba94104e180 (127.14.175.188:40999)
I20250905 19:59:21.131708 16990 raft_consensus.cc:491] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:21.131670 16494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c2a729c06ee40b39008df4cc397d7d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "865bfcc36a0b4876951dd6449b79ba9d"
I20250905 19:59:21.132521 16494 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:21.135926 16624 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c2a729c06ee40b39008df4cc397d7d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "13d4d4a7aaf346b89df6aba94104e180" is_pre_election: true
I20250905 19:59:21.136006 16623 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c2a729c06ee40b39008df4cc397d7d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "13d4d4a7aaf346b89df6aba94104e180"
I20250905 19:59:21.139604 16494 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c2a729c06ee40b39008df4cc397d7d3 in term 2.
I20250905 19:59:21.140659 16536 leader_election.cc:304] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c2a729c06ee40b39008df4cc397d7d3, 865bfcc36a0b4876951dd6449b79ba9d; no voters: 13d4d4a7aaf346b89df6aba94104e180
I20250905 19:59:21.140522 16990 raft_consensus.cc:513] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:21.141599 16494 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 2 FOLLOWER]: Rejecting Update request from peer 13d4d4a7aaf346b89df6aba94104e180 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:21.142562 16989 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:21.144052 16990 leader_election.cc:290] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159), b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:21.143709 16694 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 }, Status: INVALID_TERM, Last received: 1.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250905 19:59:21.144992 16694 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:21.145336 16694 raft_consensus.cc:738] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 13d4d4a7aaf346b89df6aba94104e180, State: Running, Role: LEADER
I20250905 19:59:21.146155 16694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:21.147161 16694 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:21.154747 16990 raft_consensus.cc:491] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0aaa68ee9814737a4ad3ab35dbbecba)
I20250905 19:59:21.155303 16990 raft_consensus.cc:513] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:21.158565 16990 leader_election.cc:290] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159), b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:21.159268 16989 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 2 LEADER]: Becoming Leader. State: Replica: 5c2a729c06ee40b39008df4cc397d7d3, State: Running, Role: LEADER
I20250905 19:59:21.160089 16989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:21.159279 16559 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "865bfcc36a0b4876951dd6449b79ba9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "5c2a729c06ee40b39008df4cc397d7d3" is_pre_election: true
I20250905 19:59:21.162324 16471 leader_election.cc:304] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 865bfcc36a0b4876951dd6449b79ba9d; no voters: 13d4d4a7aaf346b89df6aba94104e180, 5c2a729c06ee40b39008df4cc397d7d3
I20250905 19:59:21.163897 16988 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:21.164896 16558 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 2 LEADER]: Rejecting Update request from peer 13d4d4a7aaf346b89df6aba94104e180 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:21.180017 16960 consensus_queue.cc:1035] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:59:21.190609 16960 raft_consensus.cc:491] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0aaa68ee9814737a4ad3ab35dbbecba)
I20250905 19:59:21.191198 16960 raft_consensus.cc:513] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
W20250905 19:59:21.191867 16941 master_proxy_rpc.cc:203] Re-attempting GetTableSchema request to leader Master (127.14.175.188:40999)
I20250905 19:59:21.193151 16960 leader_election.cc:290] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469), 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875)
I20250905 19:59:21.197762 16751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a3f9980cc9e40efaf4855a979c04df2" is_pre_election: true
I20250905 19:59:21.199247 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" is_pre_election: true
I20250905 19:59:21.200632 16906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" is_pre_election: true
I20250905 19:59:21.206262 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" is_pre_election: true
I20250905 19:59:21.207549 16793 leader_election.cc:304] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df4471c1f39a4a5e823695f3794cc709; no voters: 3a3f9980cc9e40efaf4855a979c04df2, b0aaa68ee9814737a4ad3ab35dbbecba
I20250905 19:59:21.208195 16960 raft_consensus.cc:2747] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:21.210776 16829 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709" is_pre_election: true
I20250905 19:59:21.212436 16707 leader_election.cc:304] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3f9980cc9e40efaf4855a979c04df2; no voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709
I20250905 19:59:21.213583 16967 raft_consensus.cc:2747] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:21.214452 16828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709" is_pre_election: true
I20250905 19:59:21.215972 16707 leader_election.cc:304] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3f9980cc9e40efaf4855a979c04df2; no voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709
I20250905 19:59:21.217069 16967 raft_consensus.cc:2747] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:21.224319 16991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c2a729c06ee40b39008df4cc397d7d3. Latest consensus state: current_term: 2 leader_uuid: "5c2a729c06ee40b39008df4cc397d7d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:21.225083 16991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:21.226892 16994 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:21.233575 16994 catalog_manager.cc:671] Loaded metadata for table test-table [id=2518142617a742a6a53c69110c8d664e]
I20250905 19:59:21.239264 16994 tablet_loader.cc:96] loaded metadata for tablet 09dda0a688e441c0afdf52ec0e2a903c (table test-table [id=2518142617a742a6a53c69110c8d664e])
I20250905 19:59:21.240559 16994 tablet_loader.cc:96] loaded metadata for tablet 6183f63c34da4aa6a03ca53fff820702 (table test-table [id=2518142617a742a6a53c69110c8d664e])
I20250905 19:59:21.241899 16994 tablet_loader.cc:96] loaded metadata for tablet a9cf7a45fc234e7db1286cbf5f8f0f0b (table test-table [id=2518142617a742a6a53c69110c8d664e])
I20250905 19:59:21.243348 16994 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:21.246687 16994 catalog_manager.cc:1261] Loaded cluster ID: 84986f46af5743b49ebce666260696a5
I20250905 19:59:21.246941 16994 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:21.250114 16494 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 2 FOLLOWER]: Refusing update from remote peer 5c2a729c06ee40b39008df4cc397d7d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250905 19:59:21.250852 16994 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:21.251686 16989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250905 19:59:21.253371 16623 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 2 FOLLOWER]: Refusing update from remote peer 5c2a729c06ee40b39008df4cc397d7d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250905 19:59:21.255563 16994 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: Loaded TSK: 0
I20250905 19:59:21.255738 16989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250905 19:59:21.256778 16994 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:21.260092 16988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c2a729c06ee40b39008df4cc397d7d3. Latest consensus state: current_term: 2 leader_uuid: "5c2a729c06ee40b39008df4cc397d7d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:21.260823 16988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:21.261929 16986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c2a729c06ee40b39008df4cc397d7d3. Latest consensus state: current_term: 2 leader_uuid: "5c2a729c06ee40b39008df4cc397d7d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:21.262748 16986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:21.269150 16988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "5c2a729c06ee40b39008df4cc397d7d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:21.270146 16988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:21.272675 16991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "5c2a729c06ee40b39008df4cc397d7d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:21.273406 16991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:21.274056 16989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "5c2a729c06ee40b39008df4cc397d7d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:21.274766 16989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:21.275099 16986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "5c2a729c06ee40b39008df4cc397d7d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:21.275822 16986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:21.295295 16855 heartbeater.cc:499] Master 127.14.175.189:35421 was elected leader, sending a full tablet report...
I20250905 19:59:21.369848 16933 heartbeater.cc:499] Master 127.14.175.189:35421 was elected leader, sending a full tablet report...
I20250905 19:59:21.392776 16777 heartbeater.cc:499] Master 127.14.175.189:35421 was elected leader, sending a full tablet report...
I20250905 19:59:22.029716 16828 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b"
dest_uuid: "df4471c1f39a4a5e823695f3794cc709"
from {username='slave'} at 127.0.0.1:46748
I20250905 19:59:22.030309 16828 raft_consensus.cc:589] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 1 LEADER]: Received request to step down
I20250905 19:59:22.030591 16828 raft_consensus.cc:3053] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:22.030802 16828 raft_consensus.cc:738] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df4471c1f39a4a5e823695f3794cc709, State: Running, Role: LEADER
I20250905 19:59:22.031425 16828 consensus_queue.cc:260] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:22.032512 16828 raft_consensus.cc:3058] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:22.089915 16772 debug-util.cc:398] Leaking SignalData structure 0x7b080009a5e0 after lost signal to thread 15041
W20250905 19:59:22.090911 16772 debug-util.cc:398] Leaking SignalData structure 0x7b080008c800 after lost signal to thread 16518
W20250905 19:59:22.092151 16772 debug-util.cc:398] Leaking SignalData structure 0x7b0800233940 after lost signal to thread 16648
W20250905 19:59:22.093427 16772 debug-util.cc:398] Leaking SignalData structure 0x7b0800251ca0 after lost signal to thread 16775
I20250905 19:59:22.118173 17031 raft_consensus.cc:491] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df4471c1f39a4a5e823695f3794cc709)
I20250905 19:59:22.118728 17031 raft_consensus.cc:513] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:22.120421 17031 leader_election.cc:290] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159), b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:22.121480 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" is_pre_election: true
I20250905 19:59:22.121394 16828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709" is_pre_election: true
I20250905 19:59:22.122280 16828 raft_consensus.cc:2466] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3f9980cc9e40efaf4855a979c04df2 in term 2.
I20250905 19:59:22.123317 16707 leader_election.cc:304] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a3f9980cc9e40efaf4855a979c04df2, df4471c1f39a4a5e823695f3794cc709; no voters: b0aaa68ee9814737a4ad3ab35dbbecba
I20250905 19:59:22.123930 17031 raft_consensus.cc:2802] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:22.124279 17031 raft_consensus.cc:491] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df4471c1f39a4a5e823695f3794cc709)
I20250905 19:59:22.124541 17031 raft_consensus.cc:3058] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:22.130134 17031 raft_consensus.cc:513] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:22.131871 17031 leader_election.cc:290] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 2 election: Requested vote from peers df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159), b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:22.132654 16828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709"
I20250905 19:59:22.132810 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba"
I20250905 19:59:22.140318 16828 raft_consensus.cc:2466] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3f9980cc9e40efaf4855a979c04df2 in term 2.
I20250905 19:59:22.141426 16707 leader_election.cc:304] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a3f9980cc9e40efaf4855a979c04df2, df4471c1f39a4a5e823695f3794cc709; no voters: b0aaa68ee9814737a4ad3ab35dbbecba
I20250905 19:59:22.142096 17031 raft_consensus.cc:2802] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:22.143277 17031 raft_consensus.cc:695] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 LEADER]: Becoming Leader. State: Replica: 3a3f9980cc9e40efaf4855a979c04df2, State: Running, Role: LEADER
I20250905 19:59:22.144131 17031 consensus_queue.cc:237] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:22.155563 16541 catalog_manager.cc:5582] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 reported cstate change: term changed from 1 to 2, leader changed from df4471c1f39a4a5e823695f3794cc709 (127.14.175.130) to 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129). New cstate: current_term: 2 leader_uuid: "3a3f9980cc9e40efaf4855a979c04df2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:22.185765 16957 raft_consensus.cc:491] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df4471c1f39a4a5e823695f3794cc709)
I20250905 19:59:22.186386 16957 raft_consensus.cc:513] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:22.188463 16957 leader_election.cc:290] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875), df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159)
I20250905 19:59:22.191040 16828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709" is_pre_election: true
I20250905 19:59:22.192016 16828 raft_consensus.cc:2391] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b0aaa68ee9814737a4ad3ab35dbbecba in current term 2: Already voted for candidate 3a3f9980cc9e40efaf4855a979c04df2 in this term.
I20250905 19:59:22.191128 16751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9cf7a45fc234e7db1286cbf5f8f0f0b" candidate_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3a3f9980cc9e40efaf4855a979c04df2" is_pre_election: true
I20250905 19:59:22.194680 16871 leader_election.cc:304] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0aaa68ee9814737a4ad3ab35dbbecba; no voters: 3a3f9980cc9e40efaf4855a979c04df2, df4471c1f39a4a5e823695f3794cc709
I20250905 19:59:22.197172 16972 raft_consensus.cc:3058] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:22.205449 16972 raft_consensus.cc:2747] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:22.220535 16678 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 19:59:22.224543 16828 raft_consensus.cc:1273] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 2 FOLLOWER]: Refusing update from remote peer 3a3f9980cc9e40efaf4855a979c04df2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 19:59:22.225695 17031 consensus_queue.cc:1035] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:22.244793 16907 raft_consensus.cc:1273] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Refusing update from remote peer 3a3f9980cc9e40efaf4855a979c04df2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 19:59:22.251145 17031 consensus_queue.cc:1035] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:22.296509 16907 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702"
dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba"
from {username='slave'} at 127.0.0.1:39710
I20250905 19:59:22.297226 16907 raft_consensus.cc:589] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 LEADER]: Received request to step down
I20250905 19:59:22.297573 16907 raft_consensus.cc:3053] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:22.297869 16907 raft_consensus.cc:738] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b0aaa68ee9814737a4ad3ab35dbbecba, State: Running, Role: LEADER
I20250905 19:59:22.298749 16907 consensus_queue.cc:260] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:22.302706 16907 raft_consensus.cc:3058] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:22.412321 17000 raft_consensus.cc:491] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0aaa68ee9814737a4ad3ab35dbbecba)
I20250905 19:59:22.412940 17000 raft_consensus.cc:513] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:22.415150 17000 leader_election.cc:290] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875), b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:22.415707 16751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3a3f9980cc9e40efaf4855a979c04df2" is_pre_election: true
I20250905 19:59:22.416155 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" is_pre_election: true
I20250905 19:59:22.416932 16907 raft_consensus.cc:2466] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df4471c1f39a4a5e823695f3794cc709 in term 2.
I20250905 19:59:22.417917 16793 leader_election.cc:304] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709; no voters: 3a3f9980cc9e40efaf4855a979c04df2
I20250905 19:59:22.418566 17000 raft_consensus.cc:2802] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:22.418955 17000 raft_consensus.cc:491] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b0aaa68ee9814737a4ad3ab35dbbecba)
I20250905 19:59:22.419327 17000 raft_consensus.cc:3058] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:22.427240 17000 raft_consensus.cc:513] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:22.429399 17000 leader_election.cc:290] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 2 election: Requested vote from peers 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875), b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:22.430267 16751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3a3f9980cc9e40efaf4855a979c04df2"
I20250905 19:59:22.430563 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba"
I20250905 19:59:22.438266 16907 raft_consensus.cc:2466] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df4471c1f39a4a5e823695f3794cc709 in term 2.
I20250905 19:59:22.439330 16793 leader_election.cc:304] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709; no voters: 3a3f9980cc9e40efaf4855a979c04df2
I20250905 19:59:22.440009 17000 raft_consensus.cc:2802] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:22.442409 17000 raft_consensus.cc:695] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 2 LEADER]: Becoming Leader. State: Replica: df4471c1f39a4a5e823695f3794cc709, State: Running, Role: LEADER
I20250905 19:59:22.443284 17000 consensus_queue.cc:237] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:22.448818 17039 raft_consensus.cc:491] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0aaa68ee9814737a4ad3ab35dbbecba)
I20250905 19:59:22.449344 17039 raft_consensus.cc:513] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:22.461032 17039 leader_election.cc:290] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159), b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:22.455770 16547 catalog_manager.cc:5582] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 reported cstate change: term changed from 1 to 2, leader changed from b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131) to df4471c1f39a4a5e823695f3794cc709 (127.14.175.130). New cstate: current_term: 2 leader_uuid: "df4471c1f39a4a5e823695f3794cc709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:22.464253 16828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709" is_pre_election: true
I20250905 19:59:22.469074 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6183f63c34da4aa6a03ca53fff820702" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" is_pre_election: true
I20250905 19:59:22.470248 16907 raft_consensus.cc:2391] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a3f9980cc9e40efaf4855a979c04df2 in current term 2: Already voted for candidate df4471c1f39a4a5e823695f3794cc709 in this term.
I20250905 19:59:22.471415 16710 leader_election.cc:304] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3f9980cc9e40efaf4855a979c04df2; no voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709
I20250905 19:59:22.472839 17032 raft_consensus.cc:3058] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:22.480523 17032 raft_consensus.cc:2747] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:22.510823 16907 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c"
dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba"
from {username='slave'} at 127.0.0.1:39710
I20250905 19:59:22.511502 16907 raft_consensus.cc:589] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 LEADER]: Received request to step down
I20250905 19:59:22.511935 16907 raft_consensus.cc:3053] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:22.512265 16907 raft_consensus.cc:738] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b0aaa68ee9814737a4ad3ab35dbbecba, State: Running, Role: LEADER
I20250905 19:59:22.514380 16907 consensus_queue.cc:260] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:22.515723 16907 raft_consensus.cc:3058] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:22.530210 16907 raft_consensus.cc:1273] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Refusing update from remote peer df4471c1f39a4a5e823695f3794cc709: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 19:59:22.531320 17000 consensus_queue.cc:1035] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:22.540815 16751 raft_consensus.cc:1273] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 FOLLOWER]: Refusing update from remote peer df4471c1f39a4a5e823695f3794cc709: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 19:59:22.543011 17043 consensus_queue.cc:1035] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:22.685153 17044 raft_consensus.cc:491] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0aaa68ee9814737a4ad3ab35dbbecba)
I20250905 19:59:22.685740 17044 raft_consensus.cc:513] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:22.687726 17044 leader_election.cc:290] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469), 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875)
I20250905 19:59:22.688993 16751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3a3f9980cc9e40efaf4855a979c04df2" is_pre_election: true
I20250905 19:59:22.689249 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" is_pre_election: true
I20250905 19:59:22.689822 16907 raft_consensus.cc:2466] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df4471c1f39a4a5e823695f3794cc709 in term 2.
I20250905 19:59:22.690719 16793 leader_election.cc:304] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709; no voters: 3a3f9980cc9e40efaf4855a979c04df2
I20250905 19:59:22.691339 17044 raft_consensus.cc:2802] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:22.691675 17044 raft_consensus.cc:491] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b0aaa68ee9814737a4ad3ab35dbbecba)
I20250905 19:59:22.691926 17044 raft_consensus.cc:3058] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:22.696436 17044 raft_consensus.cc:513] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:22.697883 17044 leader_election.cc:290] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 2 election: Requested vote from peers b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469), 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875)
I20250905 19:59:22.698719 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba"
I20250905 19:59:22.698899 16751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c" candidate_uuid: "df4471c1f39a4a5e823695f3794cc709" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3a3f9980cc9e40efaf4855a979c04df2"
I20250905 19:59:22.705982 16907 raft_consensus.cc:2466] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df4471c1f39a4a5e823695f3794cc709 in term 2.
I20250905 19:59:22.707120 16793 leader_election.cc:304] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709; no voters: 3a3f9980cc9e40efaf4855a979c04df2
I20250905 19:59:22.707984 17044 raft_consensus.cc:2802] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:22.708326 17032 raft_consensus.cc:491] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0aaa68ee9814737a4ad3ab35dbbecba)
I20250905 19:59:22.708515 17044 raft_consensus.cc:695] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 2 LEADER]: Becoming Leader. State: Replica: df4471c1f39a4a5e823695f3794cc709, State: Running, Role: LEADER
I20250905 19:59:22.709082 17032 raft_consensus.cc:513] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:22.709515 17044 consensus_queue.cc:237] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } }
I20250905 19:59:22.711829 17032 leader_election.cc:290] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469), df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159)
I20250905 19:59:22.712700 16907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" is_pre_election: true
I20250905 19:59:22.713150 16828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09dda0a688e441c0afdf52ec0e2a903c" candidate_uuid: "3a3f9980cc9e40efaf4855a979c04df2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df4471c1f39a4a5e823695f3794cc709" is_pre_election: true
I20250905 19:59:22.714934 16907 raft_consensus.cc:2391] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a3f9980cc9e40efaf4855a979c04df2 in current term 2: Already voted for candidate df4471c1f39a4a5e823695f3794cc709 in this term.
I20250905 19:59:22.717906 16710 leader_election.cc:304] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3f9980cc9e40efaf4855a979c04df2; no voters: b0aaa68ee9814737a4ad3ab35dbbecba, df4471c1f39a4a5e823695f3794cc709
I20250905 19:59:22.729794 17032 raft_consensus.cc:3058] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:22.730814 16547 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49610:
table { table_id: "2518142617a742a6a53c69110c8d664e" table_name: "test-table" } new_table_name: "NewTableName"
I20250905 19:59:22.730778 16541 catalog_manager.cc:5582] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 reported cstate change: term changed from 1 to 2, leader changed from b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131) to df4471c1f39a4a5e823695f3794cc709 (127.14.175.130). New cstate: current_term: 2 leader_uuid: "df4471c1f39a4a5e823695f3794cc709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:22.743060 17032 raft_consensus.cc:2747] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:22.788731 15038 master.cc:561] Master@127.14.175.189:35421 shutting down...
I20250905 19:59:22.813375 16751 raft_consensus.cc:1273] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 FOLLOWER]: Refusing update from remote peer df4471c1f39a4a5e823695f3794cc709: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250905 19:59:22.814600 16905 raft_consensus.cc:1273] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Refusing update from remote peer df4471c1f39a4a5e823695f3794cc709: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250905 19:59:22.816236 17043 consensus_queue.cc:1035] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:22.838340 17050 tablet.cc:1722] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:59:22.852797 17041 consensus_queue.cc:1035] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:22.863229 17051 tablet.cc:1722] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:59:22.880419 17050 tablet.cc:1722] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:59:22.928268 17056 tablet.cc:1722] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:59:22.930150 17054 tablet.cc:1722] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:59:22.931864 17053 tablet.cc:1722] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:59:22.945483 17055 tablet.cc:1722] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:59:22.949972 17058 tablet.cc:1722] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:59:22.956084 17057 tablet.cc:1722] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:59:22.992041 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:59:22.993050 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:22.993484 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: stopping tablet replica
I20250905 19:59:23.022236 15038 master.cc:583] Master@127.14.175.189:35421 shutdown complete.
I20250905 19:59:23.174434 17062 raft_consensus.cc:491] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5c2a729c06ee40b39008df4cc397d7d3)
I20250905 19:59:23.174947 17062 raft_consensus.cc:513] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:23.176419 17062 leader_election.cc:290] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383), 5c2a729c06ee40b39008df4cc397d7d3 (127.14.175.189:35421)
I20250905 19:59:23.177366 16493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13d4d4a7aaf346b89df6aba94104e180" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "865bfcc36a0b4876951dd6449b79ba9d" is_pre_election: true
W20250905 19:59:23.182142 16600 leader_election.cc:336] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5c2a729c06ee40b39008df4cc397d7d3 (127.14.175.189:35421): Network error: Client connection negotiation failed: client connection to 127.14.175.189:35421: connect: Connection refused (error 111)
I20250905 19:59:23.182708 16600 leader_election.cc:304] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13d4d4a7aaf346b89df6aba94104e180; no voters: 5c2a729c06ee40b39008df4cc397d7d3, 865bfcc36a0b4876951dd6449b79ba9d
I20250905 19:59:23.183593 17062 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:23.195748 17065 raft_consensus.cc:491] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5c2a729c06ee40b39008df4cc397d7d3)
I20250905 19:59:23.196354 17065 raft_consensus.cc:513] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:23.198405 17065 leader_election.cc:290] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5c2a729c06ee40b39008df4cc397d7d3 (127.14.175.189:35421), 13d4d4a7aaf346b89df6aba94104e180 (127.14.175.188:40999)
I20250905 19:59:23.201515 16623 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "865bfcc36a0b4876951dd6449b79ba9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "13d4d4a7aaf346b89df6aba94104e180" is_pre_election: true
I20250905 19:59:23.202201 16623 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 865bfcc36a0b4876951dd6449b79ba9d in term 2.
I20250905 19:59:23.203392 16469 leader_election.cc:304] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d4d4a7aaf346b89df6aba94104e180, 865bfcc36a0b4876951dd6449b79ba9d; no voters:
I20250905 19:59:23.204087 17065 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:23.204382 17065 raft_consensus.cc:491] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5c2a729c06ee40b39008df4cc397d7d3)
I20250905 19:59:23.204645 17065 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:23.209431 16471 leader_election.cc:336] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5c2a729c06ee40b39008df4cc397d7d3 (127.14.175.189:35421): Network error: Client connection negotiation failed: client connection to 127.14.175.189:35421: connect: Connection refused (error 111)
I20250905 19:59:23.210700 17065 raft_consensus.cc:513] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:23.212152 17065 leader_election.cc:290] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [CANDIDATE]: Term 3 election: Requested vote from peers 5c2a729c06ee40b39008df4cc397d7d3 (127.14.175.189:35421), 13d4d4a7aaf346b89df6aba94104e180 (127.14.175.188:40999)
I20250905 19:59:23.213127 16623 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "865bfcc36a0b4876951dd6449b79ba9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "13d4d4a7aaf346b89df6aba94104e180"
I20250905 19:59:23.213629 16623 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:23.216176 16471 leader_election.cc:336] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 5c2a729c06ee40b39008df4cc397d7d3 (127.14.175.189:35421): Network error: Client connection negotiation failed: client connection to 127.14.175.189:35421: connect: Connection refused (error 111)
I20250905 19:59:23.219750 16623 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865bfcc36a0b4876951dd6449b79ba9d in term 3.
I20250905 19:59:23.220956 16469 leader_election.cc:304] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4d4a7aaf346b89df6aba94104e180, 865bfcc36a0b4876951dd6449b79ba9d; no voters: 5c2a729c06ee40b39008df4cc397d7d3
I20250905 19:59:23.221820 17065 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:23.223219 17065 raft_consensus.cc:695] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 3 LEADER]: Becoming Leader. State: Replica: 865bfcc36a0b4876951dd6449b79ba9d, State: Running, Role: LEADER
I20250905 19:59:23.223963 17065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:23.228014 17068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 865bfcc36a0b4876951dd6449b79ba9d. Latest consensus state: current_term: 3 leader_uuid: "865bfcc36a0b4876951dd6449b79ba9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:23.228686 17068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:23.231125 17069 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:23.237926 17069 catalog_manager.cc:671] Loaded metadata for table NewTableName [id=2518142617a742a6a53c69110c8d664e]
I20250905 19:59:23.240700 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:59:23.244235 17069 tablet_loader.cc:96] loaded metadata for tablet 09dda0a688e441c0afdf52ec0e2a903c (table NewTableName [id=2518142617a742a6a53c69110c8d664e])
I20250905 19:59:23.246284 17069 tablet_loader.cc:96] loaded metadata for tablet 6183f63c34da4aa6a03ca53fff820702 (table NewTableName [id=2518142617a742a6a53c69110c8d664e])
W20250905 19:59:23.247220 17070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:23.247972 17069 tablet_loader.cc:96] loaded metadata for tablet a9cf7a45fc234e7db1286cbf5f8f0f0b (table NewTableName [id=2518142617a742a6a53c69110c8d664e])
I20250905 19:59:23.249704 17069 catalog_manager.cc:1486] Initializing Kudu cluster ID...
W20250905 19:59:23.251148 17071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:23.256378 17069 catalog_manager.cc:1261] Loaded cluster ID: 84986f46af5743b49ebce666260696a5
I20250905 19:59:23.256836 17069 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:23.257792 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:23.260962 17073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:23.262993 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250905 19:59:23.263047 17069 catalog_manager.cc:1506] Loading token signing keys...
W20250905 19:59:23.263530 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:23.263816 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102363263791 us; error 0 us; skew 500 ppm
I20250905 19:59:23.264652 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:23.267030 17069 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: Loaded TSK: 0
I20250905 19:59:23.267875 17069 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:23.268115 15038 webserver.cc:480] Webserver started at http://127.14.175.189:39111/ using document root <none> and password file <none>
I20250905 19:59:23.268862 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:23.269101 15038 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250905 19:59:23.273844 15038 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 19:59:23.277913 17078 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:23.278914 15038 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 19:59:23.279317 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-1-root
uuid: "5c2a729c06ee40b39008df4cc397d7d3"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:23.279807 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:23.294440 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:23.295843 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:23.303723 16623 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 3 FOLLOWER]: Refusing update from remote peer 865bfcc36a0b4876951dd6449b79ba9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250905 19:59:23.307507 17068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250905 19:59:23.311259 17062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 865bfcc36a0b4876951dd6449b79ba9d. Latest consensus state: current_term: 3 leader_uuid: "865bfcc36a0b4876951dd6449b79ba9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:23.312088 17062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: This master's current role is: FOLLOWER
W20250905 19:59:23.311259 16471 consensus_peers.cc:489] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d -> Peer 5c2a729c06ee40b39008df4cc397d7d3 (127.14.175.189:35421): Couldn't send request to peer 5c2a729c06ee40b39008df4cc397d7d3. Status: Network error: Client connection negotiation failed: client connection to 127.14.175.189:35421: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:23.321060 17065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "865bfcc36a0b4876951dd6449b79ba9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:23.322075 17065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:23.328037 17062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "865bfcc36a0b4876951dd6449b79ba9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:23.328868 17062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:23.370112 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.189:35421
I20250905 19:59:23.370177 17131 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.189:35421 every 8 connection(s)
I20250905 19:59:23.380681 17132 sys_catalog.cc:263] Verifying existing consensus state
I20250905 19:59:23.387506 17132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: Bootstrap starting.
I20250905 19:59:23.399528 16935 heartbeater.cc:499] Master 127.14.175.190:38383 was elected leader, sending a full tablet report...
I20250905 19:59:23.426035 16857 heartbeater.cc:499] Master 127.14.175.190:38383 was elected leader, sending a full tablet report...
I20250905 19:59:23.445865 16778 heartbeater.cc:499] Master 127.14.175.190:38383 was elected leader, sending a full tablet report...
I20250905 19:59:23.457265 16933 heartbeater.cc:344] Connected to a master server at 127.14.175.189:35421
I20250905 19:59:23.472325 16484 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: NewTableName [id=2518142617a742a6a53c69110c8d664e] alter complete (version 1)
I20250905 19:59:23.477295 16855 heartbeater.cc:344] Connected to a master server at 127.14.175.189:35421
I20250905 19:59:23.486016 16777 heartbeater.cc:344] Connected to a master server at 127.14.175.189:35421
I20250905 19:59:23.495762 17132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:23.497102 17132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: Bootstrap complete.
I20250905 19:59:23.500241 17132 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:23.500792 17132 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c2a729c06ee40b39008df4cc397d7d3, State: Initialized, Role: FOLLOWER
I20250905 19:59:23.501428 17132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:23.503631 17140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:23.504482 17140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:23.505262 17132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:23.505679 17107 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:23.538659 17140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 865bfcc36a0b4876951dd6449b79ba9d. Latest consensus state: current_term: 3 leader_uuid: "865bfcc36a0b4876951dd6449b79ba9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:23.539557 17140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:23.538746 17150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "865bfcc36a0b4876951dd6449b79ba9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:23.540323 17150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:23.545059 17068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "865bfcc36a0b4876951dd6449b79ba9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:23.546074 17068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:23.558357 17154 catalog_manager.cc:1261] Loaded cluster ID: 84986f46af5743b49ebce666260696a5
I20250905 19:59:23.558779 17154 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: loading cluster ID for follower catalog manager: success
I20250905 19:59:23.569850 17097 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" instance_seqno: 1757102359891726) as {username='slave'} at 127.0.0.1:49800; Asking this server to re-register.
I20250905 19:59:23.571215 16933 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:23.573168 17154 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: acquiring CA information for follower catalog manager: success
I20250905 19:59:23.574041 17097 ts_manager.cc:194] Registered new tserver with Master: b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:23.582324 17154 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 19:59:23.588893 17097 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" instance_seqno: 1757102359696293) as {username='slave'} at 127.0.0.1:49810; Asking this server to re-register.
I20250905 19:59:23.590150 16855 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:23.592892 17097 ts_manager.cc:194] Registered new tserver with Master: df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159)
I20250905 19:59:23.595183 17097 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" instance_seqno: 1757102359540566) as {username='slave'} at 127.0.0.1:49812; Asking this server to re-register.
I20250905 19:59:23.596520 16777 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:23.599874 17097 ts_manager.cc:194] Registered new tserver with Master: 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875)
I20250905 19:59:23.600800 15038 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.226269894s
I20250905 19:59:23.604542 15038 master.cc:561] Master@127.14.175.190:38383 shutting down...
I20250905 19:59:23.652437 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 3 LEADER]: Raft consensus shutting down.
I20250905 19:59:23.653369 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:23.653821 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: stopping tablet replica
I20250905 19:59:23.668864 15038 master.cc:583] Master@127.14.175.190:38383 shutdown complete.
I20250905 19:59:23.748142 17150 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 865bfcc36a0b4876951dd6449b79ba9d)
I20250905 19:59:23.748678 17150 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:23.750490 17150 leader_election.cc:290] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383), 13d4d4a7aaf346b89df6aba94104e180 (127.14.175.188:40999)
W20250905 19:59:23.764118 17082 leader_election.cc:336] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383): Network error: Client connection negotiation failed: client connection to 127.14.175.190:38383: connect: Connection refused (error 111)
I20250905 19:59:23.770385 16623 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c2a729c06ee40b39008df4cc397d7d3" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "13d4d4a7aaf346b89df6aba94104e180" is_pre_election: true
I20250905 19:59:23.771556 17080 leader_election.cc:304] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c2a729c06ee40b39008df4cc397d7d3; no voters: 13d4d4a7aaf346b89df6aba94104e180, 865bfcc36a0b4876951dd6449b79ba9d
I20250905 19:59:23.772255 17150 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:23.809957 17062 raft_consensus.cc:491] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 865bfcc36a0b4876951dd6449b79ba9d)
I20250905 19:59:23.810564 17062 raft_consensus.cc:513] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:23.812461 17062 leader_election.cc:290] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383), 5c2a729c06ee40b39008df4cc397d7d3 (127.14.175.189:35421)
I20250905 19:59:23.832835 17107 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13d4d4a7aaf346b89df6aba94104e180" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "5c2a729c06ee40b39008df4cc397d7d3" is_pre_election: true
W20250905 19:59:23.833516 16600 leader_election.cc:336] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383): Network error: Client connection negotiation failed: client connection to 127.14.175.190:38383: connect: Connection refused (error 111)
I20250905 19:59:23.833571 17107 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d4d4a7aaf346b89df6aba94104e180 in term 3.
I20250905 19:59:23.834918 16600 leader_election.cc:304] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4d4a7aaf346b89df6aba94104e180, 5c2a729c06ee40b39008df4cc397d7d3; no voters: 865bfcc36a0b4876951dd6449b79ba9d
I20250905 19:59:23.837536 17062 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:23.837847 17062 raft_consensus.cc:491] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 865bfcc36a0b4876951dd6449b79ba9d)
I20250905 19:59:23.838232 17062 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:23.843912 17062 raft_consensus.cc:513] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:23.847011 17062 leader_election.cc:290] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 4 election: Requested vote from peers 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383), 5c2a729c06ee40b39008df4cc397d7d3 (127.14.175.189:35421)
I20250905 19:59:23.848727 17107 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13d4d4a7aaf346b89df6aba94104e180" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "5c2a729c06ee40b39008df4cc397d7d3"
I20250905 19:59:23.849335 17107 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:23.852792 16600 leader_election.cc:336] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383): Network error: Client connection negotiation failed: client connection to 127.14.175.190:38383: connect: Connection refused (error 111)
I20250905 19:59:23.854694 17107 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d4d4a7aaf346b89df6aba94104e180 in term 4.
I20250905 19:59:23.855590 16600 leader_election.cc:304] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4d4a7aaf346b89df6aba94104e180, 5c2a729c06ee40b39008df4cc397d7d3; no voters: 865bfcc36a0b4876951dd6449b79ba9d
I20250905 19:59:23.856330 17062 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:23.857744 17062 raft_consensus.cc:695] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 4 LEADER]: Becoming Leader. State: Replica: 13d4d4a7aaf346b89df6aba94104e180, State: Running, Role: LEADER
I20250905 19:59:23.858757 17062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:23.862524 17164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13d4d4a7aaf346b89df6aba94104e180. Latest consensus state: current_term: 4 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:23.863178 17164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:23.864840 17165 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:23.871502 17165 catalog_manager.cc:671] Loaded metadata for table NewTableName [id=2518142617a742a6a53c69110c8d664e]
I20250905 19:59:23.877692 17165 tablet_loader.cc:96] loaded metadata for tablet 09dda0a688e441c0afdf52ec0e2a903c (table NewTableName [id=2518142617a742a6a53c69110c8d664e])
I20250905 19:59:23.879259 17165 tablet_loader.cc:96] loaded metadata for tablet 6183f63c34da4aa6a03ca53fff820702 (table NewTableName [id=2518142617a742a6a53c69110c8d664e])
I20250905 19:59:23.880673 17165 tablet_loader.cc:96] loaded metadata for tablet a9cf7a45fc234e7db1286cbf5f8f0f0b (table NewTableName [id=2518142617a742a6a53c69110c8d664e])
I20250905 19:59:23.882575 17165 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:23.883965 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:59:23.887995 17165 catalog_manager.cc:1261] Loaded cluster ID: 84986f46af5743b49ebce666260696a5
I20250905 19:59:23.888362 17165 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
W20250905 19:59:23.889570 17166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:23.892067 17167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:23.894543 17165 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:23.896888 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:23.897467 17169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:23.898317 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:23.898569 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:23.898754 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102363898737 us; error 0 us; skew 500 ppm
I20250905 19:59:23.899304 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:23.900298 17165 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180: Loaded TSK: 0
I20250905 19:59:23.901432 17165 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:23.907168 15038 webserver.cc:480] Webserver started at http://127.14.175.190:46611/ using document root <none> and password file <none>
I20250905 19:59:23.907817 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:23.908075 15038 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250905 19:59:23.912385 15038 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 19:59:23.915302 17174 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:23.916133 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20250905 19:59:23.916415 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "865bfcc36a0b4876951dd6449b79ba9d"
format_stamp: "Formatted at 2025-09-05 19:59:19 on dist-test-slave-22s0"
I20250905 19:59:23.916676 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:23.936733 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:23.937945 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:23.987829 17107 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 4 FOLLOWER]: Refusing update from remote peer 13d4d4a7aaf346b89df6aba94104e180: Log matching property violated. Preceding OpId in replica: term: 3 index: 16. Preceding OpId from leader: term: 4 index: 17. (index mismatch)
I20250905 19:59:23.989753 17164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20250905 19:59:23.994553 16600 consensus_peers.cc:489] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 -> Peer 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383): Couldn't send request to peer 865bfcc36a0b4876951dd6449b79ba9d. Status: Network error: Client connection negotiation failed: client connection to 127.14.175.190:38383: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:23.996311 17150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13d4d4a7aaf346b89df6aba94104e180. Latest consensus state: current_term: 4 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:23.997356 17150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:24.002988 17062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:24.004098 17062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:24.004817 17150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:24.005661 17150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:24.016752 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:38383
I20250905 19:59:24.016927 17227 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:38383 every 8 connection(s)
I20250905 19:59:24.028554 16932 heartbeater.cc:499] Master 127.14.175.188:40999 was elected leader, sending a full tablet report...
I20250905 19:59:24.037766 16776 heartbeater.cc:499] Master 127.14.175.188:40999 was elected leader, sending a full tablet report...
I20250905 19:59:24.056506 16854 heartbeater.cc:499] Master 127.14.175.188:40999 was elected leader, sending a full tablet report...
I20250905 19:59:24.063980 17228 sys_catalog.cc:263] Verifying existing consensus state
I20250905 19:59:24.072296 16935 heartbeater.cc:344] Connected to a master server at 127.14.175.190:38383
I20250905 19:59:24.076490 17228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: Bootstrap starting.
I20250905 19:59:24.085213 16857 heartbeater.cc:344] Connected to a master server at 127.14.175.190:38383
I20250905 19:59:24.094980 16778 heartbeater.cc:344] Connected to a master server at 127.14.175.190:38383
I20250905 19:59:24.144691 17228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=7 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:24.145699 17228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: Bootstrap complete.
I20250905 19:59:24.148528 17228 raft_consensus.cc:357] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:24.149268 17228 raft_consensus.cc:738] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865bfcc36a0b4876951dd6449b79ba9d, State: Initialized, Role: FOLLOWER
I20250905 19:59:24.150004 17228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } }
I20250905 19:59:24.152047 17236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:24.152827 17236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:24.153702 17228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:24.154064 17201 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:24.188778 17246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:24.189209 17236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13d4d4a7aaf346b89df6aba94104e180. Latest consensus state: current_term: 4 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:24.189877 17246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:24.190047 17236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:59:24.196677 17191 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" instance_seqno: 1757102359696293) as {username='slave'} at 127.0.0.1:36218; Asking this server to re-register.
I20250905 19:59:24.198256 16857 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:24.200731 17250 catalog_manager.cc:1261] Loaded cluster ID: 84986f46af5743b49ebce666260696a5
I20250905 19:59:24.201040 17250 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: loading cluster ID for follower catalog manager: success
I20250905 19:59:24.202332 17191 ts_manager.cc:194] Registered new tserver with Master: df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159)
I20250905 19:59:24.203133 17190 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" instance_seqno: 1757102359540566) as {username='slave'} at 127.0.0.1:36234; Asking this server to re-register.
I20250905 19:59:24.205238 16778 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:24.208400 17250 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: acquiring CA information for follower catalog manager: success
I20250905 19:59:24.208640 17190 ts_manager.cc:194] Registered new tserver with Master: 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875)
I20250905 19:59:24.213192 17250 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 19:59:24.278671 17164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "13d4d4a7aaf346b89df6aba94104e180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865bfcc36a0b4876951dd6449b79ba9d" member_type: VOTER last_known_addr { host: "127.14.175.190" port: 38383 } } peers { permanent_uuid: "5c2a729c06ee40b39008df4cc397d7d3" member_type: VOTER last_known_addr { host: "127.14.175.189" port: 35421 } } peers { permanent_uuid: "13d4d4a7aaf346b89df6aba94104e180" member_type: VOTER last_known_addr { host: "127.14.175.188" port: 40999 } } }
I20250905 19:59:24.279500 17164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:24.286608 17190 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" instance_seqno: 1757102359891726) as {username='slave'} at 127.0.0.1:36204; Asking this server to re-register.
I20250905 19:59:24.288064 16935 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:24.291575 17190 ts_manager.cc:194] Registered new tserver with Master: b0aaa68ee9814737a4ad3ab35dbbecba (127.14.175.131:41469)
I20250905 19:59:24.291894 15038 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.263872162s
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:784: Failure
Expected equality of these values:
live_row_count
Which is: 802
table_info->GetMetrics()->live_row_count->value()
Which is: 0
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:768: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 784, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:896: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20250905 19:59:54.351497 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
W20250905 19:59:54.387519 16793 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 13 similar messages]
I20250905 19:59:54.387985 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:54.388631 15038 tablet_replica.cc:331] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2: stopping tablet replica
I20250905 19:59:54.389544 15038 raft_consensus.cc:2241] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:59:54.390810 15038 raft_consensus.cc:2270] T a9cf7a45fc234e7db1286cbf5f8f0f0b P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250905 19:59:54.392051 16793 consensus_peers.cc:489] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 -> Peer 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875): Couldn't send request to peer 3a3f9980cc9e40efaf4855a979c04df2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:54.394253 15038 tablet_replica.cc:331] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2: stopping tablet replica
I20250905 19:59:54.394797 15038 raft_consensus.cc:2241] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:54.395174 15038 raft_consensus.cc:2270] T 6183f63c34da4aa6a03ca53fff820702 P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:54.397590 15038 tablet_replica.cc:331] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2: stopping tablet replica
I20250905 19:59:54.398074 15038 raft_consensus.cc:2241] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:54.398664 15038 raft_consensus.cc:2270] T 09dda0a688e441c0afdf52ec0e2a903c P 3a3f9980cc9e40efaf4855a979c04df2 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250905 19:59:54.404999 16793 consensus_peers.cc:489] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 -> Peer 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875): Couldn't send request to peer 3a3f9980cc9e40efaf4855a979c04df2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:54.447707 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 19:59:54.462770 15038 tablet_server.cc:178] TabletServer@127.14.175.130:0 shutting down...
I20250905 19:59:54.511967 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:54.512559 15038 tablet_replica.cc:331] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709: stopping tablet replica
I20250905 19:59:54.513118 15038 raft_consensus.cc:2241] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:54.513576 15038 raft_consensus.cc:2270] T a9cf7a45fc234e7db1286cbf5f8f0f0b P df4471c1f39a4a5e823695f3794cc709 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:54.516085 15038 tablet_replica.cc:331] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709: stopping tablet replica
I20250905 19:59:54.516549 15038 raft_consensus.cc:2241] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:59:54.517210 15038 raft_consensus.cc:2270] T 6183f63c34da4aa6a03ca53fff820702 P df4471c1f39a4a5e823695f3794cc709 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:54.519608 15038 tablet_replica.cc:331] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709: stopping tablet replica
I20250905 19:59:54.520068 15038 raft_consensus.cc:2241] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:59:54.520759 15038 raft_consensus.cc:2270] T 09dda0a688e441c0afdf52ec0e2a903c P df4471c1f39a4a5e823695f3794cc709 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:54.544250 15038 tablet_server.cc:195] TabletServer@127.14.175.130:0 shutdown complete.
I20250905 19:59:54.558197 15038 tablet_server.cc:178] TabletServer@127.14.175.131:0 shutting down...
I20250905 19:59:54.567229 17299 raft_consensus.cc:491] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a3f9980cc9e40efaf4855a979c04df2)
I20250905 19:59:54.567765 17299 raft_consensus.cc:513] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3f9980cc9e40efaf4855a979c04df2" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 33875 } } peers { permanent_uuid: "df4471c1f39a4a5e823695f3794cc709" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 34159 } } peers { permanent_uuid: "b0aaa68ee9814737a4ad3ab35dbbecba" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 41469 } }
I20250905 19:59:54.570878 17299 leader_election.cc:290] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875), df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159)
W20250905 19:59:54.576925 16871 leader_election.cc:336] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3a3f9980cc9e40efaf4855a979c04df2 (127.14.175.129:33875): Network error: Client connection negotiation failed: client connection to 127.14.175.129:33875: connect: Connection refused (error 111)
W20250905 19:59:54.578770 16870 leader_election.cc:336] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df4471c1f39a4a5e823695f3794cc709 (127.14.175.130:34159): Network error: Client connection negotiation failed: client connection to 127.14.175.130:34159: connect: Connection refused (error 111)
I20250905 19:59:54.581946 16870 leader_election.cc:304] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0aaa68ee9814737a4ad3ab35dbbecba; no voters: 3a3f9980cc9e40efaf4855a979c04df2, df4471c1f39a4a5e823695f3794cc709
I20250905 19:59:54.582607 17299 raft_consensus.cc:2747] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:54.594838 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:54.595593 15038 tablet_replica.cc:331] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba: stopping tablet replica
I20250905 19:59:54.596347 15038 raft_consensus.cc:2241] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:54.597016 15038 raft_consensus.cc:2270] T a9cf7a45fc234e7db1286cbf5f8f0f0b P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:54.599750 15038 tablet_replica.cc:331] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba: stopping tablet replica
I20250905 19:59:54.600145 15038 raft_consensus.cc:2241] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:54.600505 15038 raft_consensus.cc:2270] T 6183f63c34da4aa6a03ca53fff820702 P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:54.602660 15038 tablet_replica.cc:331] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba: stopping tablet replica
I20250905 19:59:54.603065 15038 raft_consensus.cc:2241] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:54.603480 15038 raft_consensus.cc:2270] T 09dda0a688e441c0afdf52ec0e2a903c P b0aaa68ee9814737a4ad3ab35dbbecba [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:54.625471 15038 tablet_server.cc:195] TabletServer@127.14.175.131:0 shutdown complete.
I20250905 19:59:54.639605 15038 master.cc:561] Master@127.14.175.190:38383 shutting down...
I20250905 19:59:54.659431 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:54.660002 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 865bfcc36a0b4876951dd6449b79ba9d [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:54.660296 15038 tablet_replica.cc:331] stopping tablet replica
W20250905 19:59:54.665642 17178 connection.cc:441] server connection from 127.0.0.1:36240 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36240 (request call id 309) could send its response
W20250905 19:59:54.666464 16600 consensus_peers.cc:489] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 -> Peer 865bfcc36a0b4876951dd6449b79ba9d (127.14.175.190:38383): Couldn't send request to peer 865bfcc36a0b4876951dd6449b79ba9d. Status: Network error: recv got EOF from 127.14.175.190:38383 (error 108). This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:54.680003 15038 master.cc:583] Master@127.14.175.190:38383 shutdown complete.
I20250905 19:59:54.695115 15038 master.cc:561] Master@127.14.175.189:35421 shutting down...
I20250905 19:59:54.712164 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:54.712669 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 5c2a729c06ee40b39008df4cc397d7d3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:54.712988 15038 tablet_replica.cc:331] stopping tablet replica
I20250905 19:59:54.731211 15038 master.cc:583] Master@127.14.175.189:35421 shutdown complete.
I20250905 19:59:54.743155 15038 master.cc:561] Master@127.14.175.188:40999 shutting down...
I20250905 19:59:54.761603 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 4 LEADER]: Raft consensus shutting down.
I20250905 19:59:54.762571 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:54.762996 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P 13d4d4a7aaf346b89df6aba94104e180: stopping tablet replica
I20250905 19:59:54.790251 15038 master.cc:583] Master@127.14.175.188:40999 shutdown complete.
I20250905 19:59:54.822816 15038 test_util.cc:183] -----------------------------------------------
I20250905 19:59:54.823045 15038 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1757102344413625-15038-0
[ FAILED ] TsTabletManagerITest.TestTableStats (35770 ms)
[ RUN ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20250905 19:59:54.829963 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:34651
I20250905 19:59:54.831138 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:54.836447 17310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:54.837512 17311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:54.838666 17313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:54.838933 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:54.840541 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:54.840714 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:54.840833 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102394840822 us; error 0 us; skew 500 ppm
I20250905 19:59:54.841244 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:54.843559 15038 webserver.cc:480] Webserver started at http://127.14.175.190:37777/ using document root <none> and password file <none>
I20250905 19:59:54.844017 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:54.844188 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:54.844399 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:54.846004 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "f6e2a3439d8546dc90290317ef83bfd9"
format_stamp: "Formatted at 2025-09-05 19:59:54 on dist-test-slave-22s0"
I20250905 19:59:54.850705 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250905 19:59:54.853821 17318 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:54.854598 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 19:59:54.854854 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "f6e2a3439d8546dc90290317ef83bfd9"
format_stamp: "Formatted at 2025-09-05 19:59:54 on dist-test-slave-22s0"
I20250905 19:59:54.855104 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:54.873809 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:54.875007 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:54.912349 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:34651
I20250905 19:59:54.912451 17369 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:34651 every 8 connection(s)
I20250905 19:59:54.916291 17370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:54.926563 17370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9: Bootstrap starting.
I20250905 19:59:54.931430 17370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:54.935475 17370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9: No bootstrap required, opened a new log
I20250905 19:59:54.937521 17370 raft_consensus.cc:357] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2a3439d8546dc90290317ef83bfd9" member_type: VOTER }
I20250905 19:59:54.937968 17370 raft_consensus.cc:383] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:54.938207 17370 raft_consensus.cc:738] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e2a3439d8546dc90290317ef83bfd9, State: Initialized, Role: FOLLOWER
I20250905 19:59:54.938828 17370 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2a3439d8546dc90290317ef83bfd9" member_type: VOTER }
I20250905 19:59:54.939266 17370 raft_consensus.cc:397] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:54.939481 17370 raft_consensus.cc:491] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:54.939755 17370 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:54.943949 17370 raft_consensus.cc:513] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2a3439d8546dc90290317ef83bfd9" member_type: VOTER }
I20250905 19:59:54.944403 17370 leader_election.cc:304] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6e2a3439d8546dc90290317ef83bfd9; no voters:
I20250905 19:59:54.945494 17370 leader_election.cc:290] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:54.945874 17373 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:54.947222 17373 raft_consensus.cc:695] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [term 1 LEADER]: Becoming Leader. State: Replica: f6e2a3439d8546dc90290317ef83bfd9, State: Running, Role: LEADER
I20250905 19:59:54.947820 17373 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2a3439d8546dc90290317ef83bfd9" member_type: VOTER }
I20250905 19:59:54.948453 17370 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:54.951052 17374 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6e2a3439d8546dc90290317ef83bfd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2a3439d8546dc90290317ef83bfd9" member_type: VOTER } }
I20250905 19:59:54.951726 17374 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:54.951049 17375 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6e2a3439d8546dc90290317ef83bfd9. Latest consensus state: current_term: 1 leader_uuid: "f6e2a3439d8546dc90290317ef83bfd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2a3439d8546dc90290317ef83bfd9" member_type: VOTER } }
I20250905 19:59:54.952354 17375 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:54.955090 17379 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:54.959895 17379 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:54.964641 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:59:54.968663 17379 catalog_manager.cc:1349] Generated new cluster ID: a07adef5208c48af9b62eea60a955bd6
I20250905 19:59:54.968938 17379 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:54.989706 17379 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:59:54.990980 17379 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:55.003082 17379 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9: Generated new TSK 0
I20250905 19:59:55.003633 17379 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:55.031980 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:55.037966 17391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:55.039543 17392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:55.040637 17394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:55.042141 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:55.043118 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:55.043295 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:55.043416 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102395043405 us; error 0 us; skew 500 ppm
I20250905 19:59:55.043881 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:55.045949 15038 webserver.cc:480] Webserver started at http://127.14.175.129:38587/ using document root <none> and password file <none>
I20250905 19:59:55.046371 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:55.046561 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:55.046788 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:55.047904 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "90d04b4d201848d7ab80e999b82f7497"
format_stamp: "Formatted at 2025-09-05 19:59:55 on dist-test-slave-22s0"
I20250905 19:59:55.052201 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20250905 19:59:55.055251 17399 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:55.055886 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 19:59:55.056150 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "90d04b4d201848d7ab80e999b82f7497"
format_stamp: "Formatted at 2025-09-05 19:59:55 on dist-test-slave-22s0"
I20250905 19:59:55.056370 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:55.077688 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:55.078824 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:55.080140 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:55.082283 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:55.082463 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:55.082731 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:55.082887 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:55.121877 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:42293
I20250905 19:59:55.121989 17461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:42293 every 8 connection(s)
I20250905 19:59:55.126412 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:55.134331 17466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:55.135682 17467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:55.138268 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:55.138902 17469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:55.139686 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:55.139941 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:55.140156 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102395140133 us; error 0 us; skew 500 ppm
I20250905 19:59:55.140763 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:55.142932 17462 heartbeater.cc:344] Connected to a master server at 127.14.175.190:34651
I20250905 19:59:55.143275 17462 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:55.143796 15038 webserver.cc:480] Webserver started at http://127.14.175.130:35881/ using document root <none> and password file <none>
I20250905 19:59:55.144042 17462 heartbeater.cc:507] Master 127.14.175.190:34651 requested a full tablet report, sending...
I20250905 19:59:55.144327 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:55.144542 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:55.144796 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:55.146051 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-1-root/instance:
uuid: "92024a94e4ac485d9003fef3e77e6cb8"
format_stamp: "Formatted at 2025-09-05 19:59:55 on dist-test-slave-22s0"
I20250905 19:59:55.146080 17335 ts_manager.cc:194] Registered new tserver with Master: 90d04b4d201848d7ab80e999b82f7497 (127.14.175.129:42293)
I20250905 19:59:55.148138 17335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39936
I20250905 19:59:55.151087 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250905 19:59:55.154289 17474 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:55.155097 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 19:59:55.155333 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-1-root
uuid: "92024a94e4ac485d9003fef3e77e6cb8"
format_stamp: "Formatted at 2025-09-05 19:59:55 on dist-test-slave-22s0"
I20250905 19:59:55.155555 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:55.172232 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:55.173343 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:55.174747 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:55.176823 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:55.176986 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:55.177224 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:55.177351 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:55.214401 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.130:45763
I20250905 19:59:55.214511 17536 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.130:45763 every 8 connection(s)
I20250905 19:59:55.218935 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:55.225643 17540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:55.229267 17541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:55.229475 17543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:55.229766 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:55.230701 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:55.230935 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:55.231132 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102395231114 us; error 0 us; skew 500 ppm
I20250905 19:59:55.231724 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:55.231773 17537 heartbeater.cc:344] Connected to a master server at 127.14.175.190:34651
I20250905 19:59:55.232211 17537 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:55.233018 17537 heartbeater.cc:507] Master 127.14.175.190:34651 requested a full tablet report, sending...
I20250905 19:59:55.234265 15038 webserver.cc:480] Webserver started at http://127.14.175.131:41081/ using document root <none> and password file <none>
I20250905 19:59:55.235037 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:55.235076 17335 ts_manager.cc:194] Registered new tserver with Master: 92024a94e4ac485d9003fef3e77e6cb8 (127.14.175.130:45763)
I20250905 19:59:55.235316 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:55.235677 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:55.236500 17335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39938
I20250905 19:59:55.236789 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-2-root/instance:
uuid: "646e7bba6aed4bc6b8358a6aa13bba6a"
format_stamp: "Formatted at 2025-09-05 19:59:55 on dist-test-slave-22s0"
I20250905 19:59:55.241290 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20250905 19:59:55.244460 17548 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:55.245193 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:59:55.245456 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-2-root
uuid: "646e7bba6aed4bc6b8358a6aa13bba6a"
format_stamp: "Formatted at 2025-09-05 19:59:55 on dist-test-slave-22s0"
I20250905 19:59:55.245724 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:55.259737 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:55.260800 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:55.262075 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:55.264190 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:55.264375 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:55.264626 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:55.264763 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:55.302109 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.131:43061
I20250905 19:59:55.302198 17610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.131:43061 every 8 connection(s)
I20250905 19:59:55.315106 17611 heartbeater.cc:344] Connected to a master server at 127.14.175.190:34651
I20250905 19:59:55.315435 17611 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:55.316084 17611 heartbeater.cc:507] Master 127.14.175.190:34651 requested a full tablet report, sending...
I20250905 19:59:55.317862 17335 ts_manager.cc:194] Registered new tserver with Master: 646e7bba6aed4bc6b8358a6aa13bba6a (127.14.175.131:43061)
I20250905 19:59:55.318701 15038 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013458786s
I20250905 19:59:55.319180 17335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39942
I20250905 19:59:55.349866 17335 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20250905 19:59:55.351425 17335 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:59:55.392923 17427 tablet_service.cc:1468] Processing CreateTablet for tablet 5652630782914b3e92148896f484c446 (DEFAULT_TABLE table=test-table [id=fe9dc823f0834541823f494eae2c491e]), partition=
I20250905 19:59:55.394166 17427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5652630782914b3e92148896f484c446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:55.401729 17576 tablet_service.cc:1468] Processing CreateTablet for tablet 5652630782914b3e92148896f484c446 (DEFAULT_TABLE table=test-table [id=fe9dc823f0834541823f494eae2c491e]), partition=
I20250905 19:59:55.401875 17502 tablet_service.cc:1468] Processing CreateTablet for tablet 5652630782914b3e92148896f484c446 (DEFAULT_TABLE table=test-table [id=fe9dc823f0834541823f494eae2c491e]), partition=
I20250905 19:59:55.402887 17576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5652630782914b3e92148896f484c446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:55.402966 17502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5652630782914b3e92148896f484c446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:55.410180 17627 tablet_bootstrap.cc:492] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: Bootstrap starting.
I20250905 19:59:55.414705 17627 tablet_bootstrap.cc:654] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:55.418958 17629 tablet_bootstrap.cc:492] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Bootstrap starting.
I20250905 19:59:55.421013 17630 tablet_bootstrap.cc:492] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a: Bootstrap starting.
I20250905 19:59:55.422353 17627 tablet_bootstrap.cc:492] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: No bootstrap required, opened a new log
I20250905 19:59:55.422796 17627 ts_tablet_manager.cc:1397] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20250905 19:59:55.425230 17627 raft_consensus.cc:357] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d04b4d201848d7ab80e999b82f7497" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42293 } } peers { permanent_uuid: "92024a94e4ac485d9003fef3e77e6cb8" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45763 } } peers { permanent_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 43061 } }
I20250905 19:59:55.425905 17627 raft_consensus.cc:383] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:55.426198 17627 raft_consensus.cc:738] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d04b4d201848d7ab80e999b82f7497, State: Initialized, Role: FOLLOWER
I20250905 19:59:55.426656 17629 tablet_bootstrap.cc:654] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:55.426841 17627 consensus_queue.cc:260] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d04b4d201848d7ab80e999b82f7497" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42293 } } peers { permanent_uuid: "92024a94e4ac485d9003fef3e77e6cb8" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45763 } } peers { permanent_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 43061 } }
I20250905 19:59:55.427865 17630 tablet_bootstrap.cc:654] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:55.432933 17627 ts_tablet_manager.cc:1428] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
W20250905 19:59:55.433245 17627 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20250905 19:59:55.433192 17462 heartbeater.cc:499] Master 127.14.175.190:34651 was elected leader, sending a full tablet report...
I20250905 19:59:55.433993 17629 tablet_bootstrap.cc:492] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: No bootstrap required, opened a new log
I20250905 19:59:55.434093 17630 tablet_bootstrap.cc:492] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a: No bootstrap required, opened a new log
I20250905 19:59:55.434496 17629 ts_tablet_manager.cc:1397] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20250905 19:59:55.434571 17630 ts_tablet_manager.cc:1397] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20250905 19:59:55.437603 17629 raft_consensus.cc:357] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d04b4d201848d7ab80e999b82f7497" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42293 } } peers { permanent_uuid: "92024a94e4ac485d9003fef3e77e6cb8" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45763 } } peers { permanent_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 43061 } }
I20250905 19:59:55.438184 17629 raft_consensus.cc:383] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:55.438544 17629 raft_consensus.cc:738] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92024a94e4ac485d9003fef3e77e6cb8, State: Initialized, Role: FOLLOWER
I20250905 19:59:55.438079 17630 raft_consensus.cc:357] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d04b4d201848d7ab80e999b82f7497" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42293 } } peers { permanent_uuid: "92024a94e4ac485d9003fef3e77e6cb8" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45763 } } peers { permanent_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 43061 } }
I20250905 19:59:55.438992 17630 raft_consensus.cc:383] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:55.439313 17630 raft_consensus.cc:738] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646e7bba6aed4bc6b8358a6aa13bba6a, State: Initialized, Role: FOLLOWER
I20250905 19:59:55.439183 17629 consensus_queue.cc:260] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d04b4d201848d7ab80e999b82f7497" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42293 } } peers { permanent_uuid: "92024a94e4ac485d9003fef3e77e6cb8" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45763 } } peers { permanent_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 43061 } }
I20250905 19:59:55.440054 17630 consensus_queue.cc:260] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d04b4d201848d7ab80e999b82f7497" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42293 } } peers { permanent_uuid: "92024a94e4ac485d9003fef3e77e6cb8" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45763 } } peers { permanent_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 43061 } }
I20250905 19:59:55.441322 17537 heartbeater.cc:499] Master 127.14.175.190:34651 was elected leader, sending a full tablet report...
I20250905 19:59:55.441895 17629 ts_tablet_manager.cc:1428] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20250905 19:59:55.442183 17629 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20250905 19:59:55.442451 17611 heartbeater.cc:499] Master 127.14.175.190:34651 was elected leader, sending a full tablet report...
I20250905 19:59:55.443140 17630 ts_tablet_manager.cc:1428] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20250905 19:59:55.443425 17630 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20250905 19:59:55.443583 17631 raft_consensus.cc:491] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:55.443979 17631 raft_consensus.cc:513] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d04b4d201848d7ab80e999b82f7497" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42293 } } peers { permanent_uuid: "92024a94e4ac485d9003fef3e77e6cb8" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45763 } } peers { permanent_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 43061 } }
I20250905 19:59:55.449308 17631 leader_election.cc:290] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92024a94e4ac485d9003fef3e77e6cb8 (127.14.175.130:45763), 646e7bba6aed4bc6b8358a6aa13bba6a (127.14.175.131:43061)
I20250905 19:59:55.456588 17512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5652630782914b3e92148896f484c446" candidate_uuid: "90d04b4d201848d7ab80e999b82f7497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92024a94e4ac485d9003fef3e77e6cb8" is_pre_election: true
I20250905 19:59:55.457355 17512 raft_consensus.cc:2466] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d04b4d201848d7ab80e999b82f7497 in term 0.
I20250905 19:59:55.458282 17586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5652630782914b3e92148896f484c446" candidate_uuid: "90d04b4d201848d7ab80e999b82f7497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" is_pre_election: true
I20250905 19:59:55.458482 17402 leader_election.cc:304] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90d04b4d201848d7ab80e999b82f7497, 92024a94e4ac485d9003fef3e77e6cb8; no voters:
I20250905 19:59:55.458849 17586 raft_consensus.cc:2466] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d04b4d201848d7ab80e999b82f7497 in term 0.
I20250905 19:59:55.459256 17631 raft_consensus.cc:2802] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:59:55.459600 17631 raft_consensus.cc:491] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:55.459944 17631 raft_consensus.cc:3058] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:55.464794 17631 raft_consensus.cc:513] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d04b4d201848d7ab80e999b82f7497" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42293 } } peers { permanent_uuid: "92024a94e4ac485d9003fef3e77e6cb8" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45763 } } peers { permanent_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 43061 } }
I20250905 19:59:55.466166 17631 leader_election.cc:290] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [CANDIDATE]: Term 1 election: Requested vote from peers 92024a94e4ac485d9003fef3e77e6cb8 (127.14.175.130:45763), 646e7bba6aed4bc6b8358a6aa13bba6a (127.14.175.131:43061)
I20250905 19:59:55.466822 17512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5652630782914b3e92148896f484c446" candidate_uuid: "90d04b4d201848d7ab80e999b82f7497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92024a94e4ac485d9003fef3e77e6cb8"
I20250905 19:59:55.467048 17586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5652630782914b3e92148896f484c446" candidate_uuid: "90d04b4d201848d7ab80e999b82f7497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a"
I20250905 19:59:55.467321 17512 raft_consensus.cc:3058] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:55.467509 17586 raft_consensus.cc:3058] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:59:55.468008 17538 tablet.cc:2378] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 19:59:55.473945 17512 raft_consensus.cc:2466] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d04b4d201848d7ab80e999b82f7497 in term 1.
I20250905 19:59:55.473948 17586 raft_consensus.cc:2466] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d04b4d201848d7ab80e999b82f7497 in term 1.
I20250905 19:59:55.475190 17402 leader_election.cc:304] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90d04b4d201848d7ab80e999b82f7497, 92024a94e4ac485d9003fef3e77e6cb8; no voters:
I20250905 19:59:55.475849 17631 raft_consensus.cc:2802] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:55.476634 17631 raft_consensus.cc:695] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 1 LEADER]: Becoming Leader. State: Replica: 90d04b4d201848d7ab80e999b82f7497, State: Running, Role: LEADER
I20250905 19:59:55.477387 17631 consensus_queue.cc:237] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d04b4d201848d7ab80e999b82f7497" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42293 } } peers { permanent_uuid: "92024a94e4ac485d9003fef3e77e6cb8" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45763 } } peers { permanent_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 43061 } }
I20250905 19:59:55.484548 17335 catalog_manager.cc:5582] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d04b4d201848d7ab80e999b82f7497 (127.14.175.129). New cstate: current_term: 1 leader_uuid: "90d04b4d201848d7ab80e999b82f7497" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d04b4d201848d7ab80e999b82f7497" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92024a94e4ac485d9003fef3e77e6cb8" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 43061 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:55.523875 17502 tablet_service.cc:1515] Processing DeleteTablet for tablet 5652630782914b3e92148896f484c446 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:49366
I20250905 19:59:55.525211 17640 tablet_replica.cc:331] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: stopping tablet replica
I20250905 19:59:55.525801 17640 raft_consensus.cc:2241] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:55.526212 17640 raft_consensus.cc:2270] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:55.528339 17640 ts_tablet_manager.cc:1905] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:55.537976 17640 ts_tablet_manager.cc:1918] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 0.0
I20250905 19:59:55.538282 17640 log.cc:1199] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Deleting WAL directory at /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-1-root/wals/5652630782914b3e92148896f484c446
I20250905 19:59:55.541971 17641 ts_tablet_manager.cc:927] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Initiating tablet copy from peer 90d04b4d201848d7ab80e999b82f7497 (127.14.175.129:42293)
I20250905 19:59:55.543740 17641 tablet_copy_client.cc:323] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: tablet copy: Beginning tablet copy session from remote peer at address 127.14.175.129:42293
I20250905 19:59:55.551309 17447 tablet_copy_service.cc:140] P 90d04b4d201848d7ab80e999b82f7497: Received BeginTabletCopySession request for tablet 5652630782914b3e92148896f484c446 from peer 92024a94e4ac485d9003fef3e77e6cb8 ({username='slave'} at 127.0.0.1:45852)
I20250905 19:59:55.551781 17447 tablet_copy_service.cc:161] P 90d04b4d201848d7ab80e999b82f7497: Beginning new tablet copy session on tablet 5652630782914b3e92148896f484c446 from peer 92024a94e4ac485d9003fef3e77e6cb8 at {username='slave'} at 127.0.0.1:45852: session id = 92024a94e4ac485d9003fef3e77e6cb8-5652630782914b3e92148896f484c446
I20250905 19:59:55.557224 17447 tablet_copy_source_session.cc:215] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:55.559432 17641 ts_tablet_manager.cc:1905] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250905 19:59:55.568554 17641 ts_tablet_manager.cc:1918] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 0.0
I20250905 19:59:55.569082 17641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5652630782914b3e92148896f484c446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:55.573582 17641 tablet_copy_client.cc:806] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:55.574019 17641 tablet_copy_client.cc:670] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:55.575814 17333 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39958:
table { table_id: "fe9dc823f0834541823f494eae2c491e" table_name: "test-table" }
I20250905 19:59:55.576453 17333 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39958:
table { table_id: "fe9dc823f0834541823f494eae2c491e" table_name: "test-table" }
I20250905 19:59:55.577751 17641 tablet_copy_client.cc:538] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:55.583050 17641 tablet_bootstrap.cc:492] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Bootstrap starting.
I20250905 19:59:55.585896 17333 catalog_manager.cc:5869] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9: Sending DeleteTablet for 3 replicas of tablet 5652630782914b3e92148896f484c446
I20250905 19:59:55.587646 17427 tablet_service.cc:1515] Processing DeleteTablet for tablet 5652630782914b3e92148896f484c446 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:59:55 UTC) from {username='slave'} at 127.0.0.1:45836
I20250905 19:59:55.588011 17502 tablet_service.cc:1515] Processing DeleteTablet for tablet 5652630782914b3e92148896f484c446 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:59:55 UTC) from {username='slave'} at 127.0.0.1:49336
I20250905 19:59:55.588393 17576 tablet_service.cc:1515] Processing DeleteTablet for tablet 5652630782914b3e92148896f484c446 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:59:55 UTC) from {username='slave'} at 127.0.0.1:50350
W20250905 19:59:55.589973 17321 catalog_manager.cc:4908] TS 92024a94e4ac485d9003fef3e77e6cb8 (127.14.175.130:45763): delete failed for tablet 5652630782914b3e92148896f484c446 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5652630782914b3e92148896f484c446 already in progress: copying tablet
I20250905 19:59:55.591876 17644 tablet_replica.cc:331] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a: stopping tablet replica
I20250905 19:59:55.592805 17643 tablet_replica.cc:331] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: stopping tablet replica
I20250905 19:59:55.592763 17644 raft_consensus.cc:2241] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:55.593629 17644 raft_consensus.cc:2270] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:55.593796 17643 raft_consensus.cc:2241] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:55.594714 17643 pending_rounds.cc:70] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: Trying to abort 1 pending ops.
I20250905 19:59:55.595153 17643 pending_rounds.cc:77] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197091411881472000 op_type: NO_OP noop_request { }
I20250905 19:59:55.595664 17643 raft_consensus.cc:2887] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:59:55.596102 17643 raft_consensus.cc:2270] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:55.596563 17644 ts_tablet_manager.cc:1905] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:59:55.598664 17643 ts_tablet_manager.cc:1905] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:59:55.599354 17641 tablet_bootstrap.cc:492] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250905 19:59:55.600255 17641 tablet_bootstrap.cc:492] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Bootstrap complete.
I20250905 19:59:55.600878 17641 ts_tablet_manager.cc:1397] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20250905 19:59:55.603732 17641 raft_consensus.cc:357] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d04b4d201848d7ab80e999b82f7497" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42293 } } peers { permanent_uuid: "92024a94e4ac485d9003fef3e77e6cb8" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45763 } } peers { permanent_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 43061 } }
I20250905 19:59:55.604619 17641 raft_consensus.cc:738] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92024a94e4ac485d9003fef3e77e6cb8, State: Initialized, Role: FOLLOWER
I20250905 19:59:55.605459 17641 consensus_queue.cc:260] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d04b4d201848d7ab80e999b82f7497" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42293 } } peers { permanent_uuid: "92024a94e4ac485d9003fef3e77e6cb8" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 45763 } } peers { permanent_uuid: "646e7bba6aed4bc6b8358a6aa13bba6a" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 43061 } }
I20250905 19:59:55.607621 17641 ts_tablet_manager.cc:1428] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20250905 19:59:55.607949 17641 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20250905 19:59:55.610373 17643 ts_tablet_manager.cc:1918] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:59:55.610842 17643 log.cc:1199] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: Deleting WAL directory at /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-0-root/wals/5652630782914b3e92148896f484c446
I20250905 19:59:55.610951 17644 ts_tablet_manager.cc:1918] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:59:55.611467 17644 log.cc:1199] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a: Deleting WAL directory at /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-2-root/wals/5652630782914b3e92148896f484c446
I20250905 19:59:55.611896 17643 ts_tablet_manager.cc:1939] T 5652630782914b3e92148896f484c446 P 90d04b4d201848d7ab80e999b82f7497: Deleting consensus metadata
I20250905 19:59:55.612159 17644 ts_tablet_manager.cc:1939] T 5652630782914b3e92148896f484c446 P 646e7bba6aed4bc6b8358a6aa13bba6a: Deleting consensus metadata
I20250905 19:59:55.612900 17502 tablet_service.cc:1515] Processing DeleteTablet for tablet 5652630782914b3e92148896f484c446 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:59:55 UTC) from {username='slave'} at 127.0.0.1:49336
I20250905 19:59:55.613438 17321 catalog_manager.cc:4928] TS 90d04b4d201848d7ab80e999b82f7497 (127.14.175.129:42293): tablet 5652630782914b3e92148896f484c446 (table test-table [id=fe9dc823f0834541823f494eae2c491e]) successfully deleted
W20250905 19:59:55.614284 17321 catalog_manager.cc:4908] TS 92024a94e4ac485d9003fef3e77e6cb8 (127.14.175.130:45763): delete failed for tablet 5652630782914b3e92148896f484c446 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5652630782914b3e92148896f484c446 already in progress: copying tablet
I20250905 19:59:55.615139 17319 catalog_manager.cc:4928] TS 646e7bba6aed4bc6b8358a6aa13bba6a (127.14.175.131:43061): tablet 5652630782914b3e92148896f484c446 (table test-table [id=fe9dc823f0834541823f494eae2c491e]) successfully deleted
I20250905 19:59:55.660064 17447 tablet_copy_service.cc:342] P 90d04b4d201848d7ab80e999b82f7497: Request end of tablet copy session 92024a94e4ac485d9003fef3e77e6cb8-5652630782914b3e92148896f484c446 received from {username='slave'} at 127.0.0.1:45852
I20250905 19:59:55.660511 17447 tablet_copy_service.cc:434] P 90d04b4d201848d7ab80e999b82f7497: ending tablet copy session 92024a94e4ac485d9003fef3e77e6cb8-5652630782914b3e92148896f484c446 on tablet 5652630782914b3e92148896f484c446 with peer 92024a94e4ac485d9003fef3e77e6cb8
I20250905 19:59:55.663569 17502 tablet_service.cc:1515] Processing DeleteTablet for tablet 5652630782914b3e92148896f484c446 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:59:55 UTC) from {username='slave'} at 127.0.0.1:49336
I20250905 19:59:55.664201 17640 tablet_replica.cc:331] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: stopping tablet replica
I20250905 19:59:55.664744 17640 raft_consensus.cc:2241] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:55.665103 17640 pending_rounds.cc:70] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Trying to abort 1 pending ops.
I20250905 19:59:55.665285 17640 pending_rounds.cc:77] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197091411881472000 op_type: NO_OP noop_request { }
I20250905 19:59:55.665563 17640 raft_consensus.cc:2887] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:59:55.665871 17640 raft_consensus.cc:2270] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:55.667877 17640 ts_tablet_manager.cc:1905] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:59:55.677418 17640 ts_tablet_manager.cc:1918] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:59:55.677691 17640 log.cc:1199] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Deleting WAL directory at /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1757102344413625-15038-0/minicluster-data/ts-1-root/wals/5652630782914b3e92148896f484c446
I20250905 19:59:55.678241 17640 ts_tablet_manager.cc:1939] T 5652630782914b3e92148896f484c446 P 92024a94e4ac485d9003fef3e77e6cb8: Deleting consensus metadata
I20250905 19:59:55.680326 17321 catalog_manager.cc:4928] TS 92024a94e4ac485d9003fef3e77e6cb8 (127.14.175.130:45763): tablet 5652630782914b3e92148896f484c446 (table test-table [id=fe9dc823f0834541823f494eae2c491e]) successfully deleted
I20250905 19:59:55.726157 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
I20250905 19:59:55.742349 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:55.749279 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 19:59:55.757603 15038 tablet_server.cc:178] TabletServer@127.14.175.130:0 shutting down...
I20250905 19:59:55.772377 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:55.788408 15038 tablet_server.cc:195] TabletServer@127.14.175.130:0 shutdown complete.
I20250905 19:59:55.797016 15038 tablet_server.cc:178] TabletServer@127.14.175.131:0 shutting down...
I20250905 19:59:55.811792 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:55.827541 15038 tablet_server.cc:195] TabletServer@127.14.175.131:0 shutdown complete.
I20250905 19:59:55.835420 15038 master.cc:561] Master@127.14.175.190:34651 shutting down...
I20250905 19:59:55.849352 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:55.849875 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:55.850215 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P f6e2a3439d8546dc90290317ef83bfd9: stopping tablet replica
I20250905 19:59:55.868394 15038 master.cc:583] Master@127.14.175.190:34651 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1062 ms)
[----------] 4 tests from TsTabletManagerITest (41411 ms total)
[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20250905 19:59:55.891938 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:34001
I20250905 19:59:55.892935 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:55.897270 17650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:55.898608 17651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:55.899588 17653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:55.899781 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:55.900740 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:55.900928 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:55.901044 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102395901033 us; error 0 us; skew 500 ppm
I20250905 19:59:55.901470 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:55.903569 15038 webserver.cc:480] Webserver started at http://127.14.175.190:37797/ using document root <none> and password file <none>
I20250905 19:59:55.903995 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:55.904139 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:55.904338 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:55.905339 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "eabd25c9d71648a38495f7edcb8c0353"
format_stamp: "Formatted at 2025-09-05 19:59:55 on dist-test-slave-22s0"
I20250905 19:59:55.909446 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 19:59:55.912509 17658 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:55.913259 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:59:55.913513 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "eabd25c9d71648a38495f7edcb8c0353"
format_stamp: "Formatted at 2025-09-05 19:59:55 on dist-test-slave-22s0"
I20250905 19:59:55.913779 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:55.933272 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:55.934227 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:55.971170 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:34001
I20250905 19:59:55.971251 17709 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:34001 every 8 connection(s)
I20250905 19:59:55.974661 17710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:55.984753 17710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353: Bootstrap starting.
I20250905 19:59:55.988682 17710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:55.992254 17710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353: No bootstrap required, opened a new log
I20250905 19:59:55.994021 17710 raft_consensus.cc:357] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eabd25c9d71648a38495f7edcb8c0353" member_type: VOTER }
I20250905 19:59:55.994395 17710 raft_consensus.cc:383] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:55.994633 17710 raft_consensus.cc:738] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eabd25c9d71648a38495f7edcb8c0353, State: Initialized, Role: FOLLOWER
I20250905 19:59:55.995182 17710 consensus_queue.cc:260] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eabd25c9d71648a38495f7edcb8c0353" member_type: VOTER }
I20250905 19:59:55.995611 17710 raft_consensus.cc:397] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:55.995862 17710 raft_consensus.cc:491] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:55.996109 17710 raft_consensus.cc:3058] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:55.999830 17710 raft_consensus.cc:513] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eabd25c9d71648a38495f7edcb8c0353" member_type: VOTER }
I20250905 19:59:56.000270 17710 leader_election.cc:304] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eabd25c9d71648a38495f7edcb8c0353; no voters:
I20250905 19:59:56.001260 17710 leader_election.cc:290] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:56.001507 17713 raft_consensus.cc:2802] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:56.002755 17713 raft_consensus.cc:695] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [term 1 LEADER]: Becoming Leader. State: Replica: eabd25c9d71648a38495f7edcb8c0353, State: Running, Role: LEADER
I20250905 19:59:56.003384 17713 consensus_queue.cc:237] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eabd25c9d71648a38495f7edcb8c0353" member_type: VOTER }
I20250905 19:59:56.004014 17710 sys_catalog.cc:564] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:56.007218 17714 sys_catalog.cc:455] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eabd25c9d71648a38495f7edcb8c0353" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eabd25c9d71648a38495f7edcb8c0353" member_type: VOTER } }
I20250905 19:59:56.007784 17714 sys_catalog.cc:458] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:56.008442 17715 sys_catalog.cc:455] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eabd25c9d71648a38495f7edcb8c0353. Latest consensus state: current_term: 1 leader_uuid: "eabd25c9d71648a38495f7edcb8c0353" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eabd25c9d71648a38495f7edcb8c0353" member_type: VOTER } }
I20250905 19:59:56.009020 17715 sys_catalog.cc:458] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:56.011129 17720 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:56.019019 17720 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:56.019938 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:59:56.027170 17720 catalog_manager.cc:1349] Generated new cluster ID: 874dcf1dd6754371b448c6759ed00c8a
I20250905 19:59:56.027439 17720 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:56.060050 17720 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:59:56.061261 17720 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:56.088996 17720 catalog_manager.cc:5955] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353: Generated new TSK 0
I20250905 19:59:56.089569 17720 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:56.150811 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:56.156124 17731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:56.157425 17732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:56.159166 17734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:56.159296 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:56.160105 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:56.160293 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:56.160441 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102396160422 us; error 0 us; skew 500 ppm
I20250905 19:59:56.160907 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:56.162998 15038 webserver.cc:480] Webserver started at http://127.14.175.129:37097/ using document root <none> and password file <none>
I20250905 19:59:56.163473 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:56.163657 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:56.163911 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:56.164955 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679"
format_stamp: "Formatted at 2025-09-05 19:59:56 on dist-test-slave-22s0"
I20250905 19:59:56.168982 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 19:59:56.171895 17739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:56.172513 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 19:59:56.172765 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679"
format_stamp: "Formatted at 2025-09-05 19:59:56 on dist-test-slave-22s0"
I20250905 19:59:56.173017 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:56.181947 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:56.182952 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:56.187521 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:56.187852 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:56.188184 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:56.188441 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:56.239776 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:37871
I20250905 19:59:56.239868 17809 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:37871 every 8 connection(s)
I20250905 19:59:56.244045 17810 heartbeater.cc:344] Connected to a master server at 127.14.175.190:34001
I20250905 19:59:56.244370 17810 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:56.245040 17810 heartbeater.cc:507] Master 127.14.175.190:34001 requested a full tablet report, sending...
I20250905 19:59:56.246723 17675 ts_manager.cc:194] Registered new tserver with Master: bb6f54ef86ae48fdbe35c7d2af7b0679 (127.14.175.129:37871)
I20250905 19:59:56.247534 15038 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004753059s
I20250905 19:59:56.248376 17675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58286
I20250905 19:59:56.276769 17775 tablet_service.cc:1468] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20250905 19:59:56.277920 17775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:56.288758 17820 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: Bootstrap starting.
I20250905 19:59:56.292711 17820 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:56.296283 17820 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: No bootstrap required, opened a new log
I20250905 19:59:56.296619 17820 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20250905 19:59:56.298347 17820 raft_consensus.cc:357] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37871 } }
I20250905 19:59:56.298744 17820 raft_consensus.cc:383] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:56.298923 17820 raft_consensus.cc:738] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6f54ef86ae48fdbe35c7d2af7b0679, State: Initialized, Role: FOLLOWER
I20250905 19:59:56.299384 17820 consensus_queue.cc:260] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37871 } }
I20250905 19:59:56.299777 17820 raft_consensus.cc:397] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:56.299971 17820 raft_consensus.cc:491] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:56.300220 17820 raft_consensus.cc:3058] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:56.304638 17820 raft_consensus.cc:513] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37871 } }
I20250905 19:59:56.305119 17820 leader_election.cc:304] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb6f54ef86ae48fdbe35c7d2af7b0679; no voters:
I20250905 19:59:56.306226 17820 leader_election.cc:290] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:56.306541 17822 raft_consensus.cc:2802] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:56.307879 17822 raft_consensus.cc:695] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 1 LEADER]: Becoming Leader. State: Replica: bb6f54ef86ae48fdbe35c7d2af7b0679, State: Running, Role: LEADER
I20250905 19:59:56.308041 17810 heartbeater.cc:499] Master 127.14.175.190:34001 was elected leader, sending a full tablet report...
I20250905 19:59:56.308657 17820 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20250905 19:59:56.308686 17822 consensus_queue.cc:237] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37871 } }
I20250905 19:59:56.313246 17823 tablet_replica.cc:440] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" committed_config { opid_index: -1 peers { permanent_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37871 } } }
I20250905 19:59:56.313156 17824 tablet_replica.cc:440] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: TxnStatusTablet state changed. Reason: New leader bb6f54ef86ae48fdbe35c7d2af7b0679. Latest consensus state: current_term: 1 leader_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" committed_config { opid_index: -1 peers { permanent_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37871 } } }
I20250905 19:59:56.314070 17823 tablet_replica.cc:443] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: This TxnStatusTablet replica's current role is: LEADER
I20250905 19:59:56.314231 17824 tablet_replica.cc:443] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: This TxnStatusTablet replica's current role is: LEADER
W20250905 19:59:56.316390 17675 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250905 19:59:56.316629 17827 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250905 19:59:56.317071 17827 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250905 19:59:56.320569 17675 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250905 19:59:56.353399 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
I20250905 19:59:56.372990 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:56.373688 15038 tablet_replica.cc:331] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: stopping tablet replica
I20250905 19:59:56.374310 15038 raft_consensus.cc:2241] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:56.374923 15038 raft_consensus.cc:2270] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:56.386008 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 19:59:56.401523 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:56.408315 17830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:56.408825 17831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:56.411440 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:56.412233 17833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:56.413295 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:56.413550 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:56.413744 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102396413726 us; error 0 us; skew 500 ppm
I20250905 19:59:56.414530 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:56.417237 15038 webserver.cc:480] Webserver started at http://127.14.175.129:37097/ using document root <none> and password file <none>
I20250905 19:59:56.417883 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:56.418118 15038 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250905 19:59:56.422977 15038 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250905 19:59:56.426627 17838 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:56.427445 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 19:59:56.427780 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679"
format_stamp: "Formatted at 2025-09-05 19:59:56 on dist-test-slave-22s0"
I20250905 19:59:56.428165 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:56.446763 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:56.448146 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:56.461126 17847 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 19:59:56.469739 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 19:59:56.470062 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.003s sys 0.003s
I20250905 19:59:56.470366 15038 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 19:59:56.477069 17847 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: Bootstrap starting.
I20250905 19:59:56.477437 15038 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 19:59:56.477921 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.007s sys 0.000s
I20250905 19:59:56.513906 17847 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:56.515072 17847 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: Bootstrap complete.
I20250905 19:59:56.515703 17847 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: Time spent bootstrapping tablet: real 0.039s user 0.029s sys 0.008s
I20250905 19:59:56.518175 17847 raft_consensus.cc:357] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37871 } }
I20250905 19:59:56.518852 17847 raft_consensus.cc:738] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6f54ef86ae48fdbe35c7d2af7b0679, State: Initialized, Role: FOLLOWER
I20250905 19:59:56.519580 17847 consensus_queue.cc:260] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37871 } }
I20250905 19:59:56.520288 17847 raft_consensus.cc:397] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:56.520623 17847 raft_consensus.cc:491] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:56.521016 17847 raft_consensus.cc:3058] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:56.528136 17847 raft_consensus.cc:513] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37871 } }
I20250905 19:59:56.528928 17847 leader_election.cc:304] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb6f54ef86ae48fdbe35c7d2af7b0679; no voters:
I20250905 19:59:56.530684 17847 leader_election.cc:290] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 19:59:56.531113 17882 raft_consensus.cc:2802] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:56.535252 17847 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
I20250905 19:59:56.538937 17882 raft_consensus.cc:695] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 2 LEADER]: Becoming Leader. State: Replica: bb6f54ef86ae48fdbe35c7d2af7b0679, State: Running, Role: LEADER
I20250905 19:59:56.539996 17882 consensus_queue.cc:237] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37871 } }
I20250905 19:59:56.556918 17885 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: New leader bb6f54ef86ae48fdbe35c7d2af7b0679. Latest consensus state: current_term: 2 leader_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" committed_config { opid_index: -1 peers { permanent_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37871 } } }
I20250905 19:59:56.557667 17885 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: LEADER
I20250905 19:59:56.565991 17884 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" committed_config { opid_index: -1 peers { permanent_uuid: "bb6f54ef86ae48fdbe35c7d2af7b0679" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37871 } } }
I20250905 19:59:56.566744 17884 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: LEADER
I20250905 19:59:56.568807 17906 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250905 19:59:56.569209 17906 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250905 19:59:56.579229 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:37871
I20250905 19:59:56.579293 17915 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:37871 every 8 connection(s)
I20250905 19:59:56.585419 17916 heartbeater.cc:344] Connected to a master server at 127.14.175.190:34001
I20250905 19:59:56.585878 17916 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:56.586771 17916 heartbeater.cc:507] Master 127.14.175.190:34001 requested a full tablet report, sending...
I20250905 19:59:56.588239 15038 tablet_server.cc:178] TabletServer@127.14.175.129:37871 shutting down...
I20250905 19:59:56.592911 17675 ts_manager.cc:194] Re-registered known tserver with Master: bb6f54ef86ae48fdbe35c7d2af7b0679 (127.14.175.129:37871)
W20250905 19:59:56.593629 17675 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250905 19:59:56.595443 17675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58322
I20250905 19:59:56.598734 17916 heartbeater.cc:499] Master 127.14.175.190:34001 was elected leader, sending a full tablet report...
W20250905 19:59:56.601964 17675 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250905 19:59:56.609597 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:56.610280 15038 tablet_replica.cc:331] stopping tablet replica
I20250905 19:59:56.610993 15038 raft_consensus.cc:2241] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:59:56.611449 15038 raft_consensus.cc:2270] T 11111111111111111111111111111111 P bb6f54ef86ae48fdbe35c7d2af7b0679 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:56.623023 15038 tablet_server.cc:195] TabletServer@127.14.175.129:37871 shutdown complete.
I20250905 19:59:56.636183 15038 master.cc:561] Master@127.14.175.190:34001 shutting down...
I20250905 19:59:56.660475 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:56.661121 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:56.661496 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P eabd25c9d71648a38495f7edcb8c0353: stopping tablet replica
I20250905 19:59:56.680915 15038 master.cc:583] Master@127.14.175.190:34001 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (819 ms)
[ RUN ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20250905 19:59:56.713621 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:45399
I20250905 19:59:56.715071 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:56.720554 17922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:56.721483 17923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:56.723855 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:56.724716 17926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:56.725565 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:56.725826 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:56.726020 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102396726002 us; error 0 us; skew 500 ppm
I20250905 19:59:56.726817 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:56.729542 15038 webserver.cc:480] Webserver started at http://127.14.175.190:37885/ using document root <none> and password file <none>
I20250905 19:59:56.730156 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:56.730374 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:56.730702 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:56.732144 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "d0809eccf0c14cfe89df1c01b7dbb575"
format_stamp: "Formatted at 2025-09-05 19:59:56 on dist-test-slave-22s0"
I20250905 19:59:56.737846 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20250905 19:59:56.742180 17930 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:56.743006 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 19:59:56.743330 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "d0809eccf0c14cfe89df1c01b7dbb575"
format_stamp: "Formatted at 2025-09-05 19:59:56 on dist-test-slave-22s0"
I20250905 19:59:56.743674 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:56.760037 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:56.760995 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:56.801433 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:45399
I20250905 19:59:56.801577 17981 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:45399 every 8 connection(s)
I20250905 19:59:56.805559 17982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:56.820618 17982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575: Bootstrap starting.
I20250905 19:59:56.826797 17982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:56.831511 17982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575: No bootstrap required, opened a new log
I20250905 19:59:56.834241 17982 raft_consensus.cc:357] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0809eccf0c14cfe89df1c01b7dbb575" member_type: VOTER }
I20250905 19:59:56.834802 17982 raft_consensus.cc:383] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:56.835114 17982 raft_consensus.cc:738] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0809eccf0c14cfe89df1c01b7dbb575, State: Initialized, Role: FOLLOWER
I20250905 19:59:56.835747 17982 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0809eccf0c14cfe89df1c01b7dbb575" member_type: VOTER }
I20250905 19:59:56.836337 17982 raft_consensus.cc:397] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:56.836623 17982 raft_consensus.cc:491] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:56.836931 17982 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:56.843706 17982 raft_consensus.cc:513] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0809eccf0c14cfe89df1c01b7dbb575" member_type: VOTER }
I20250905 19:59:56.844334 17982 leader_election.cc:304] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0809eccf0c14cfe89df1c01b7dbb575; no voters:
I20250905 19:59:56.845710 17982 leader_election.cc:290] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:56.846160 17985 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:56.847945 17985 raft_consensus.cc:695] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [term 1 LEADER]: Becoming Leader. State: Replica: d0809eccf0c14cfe89df1c01b7dbb575, State: Running, Role: LEADER
I20250905 19:59:56.848609 17985 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0809eccf0c14cfe89df1c01b7dbb575" member_type: VOTER }
I20250905 19:59:56.849150 17982 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:56.851270 17986 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0809eccf0c14cfe89df1c01b7dbb575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0809eccf0c14cfe89df1c01b7dbb575" member_type: VOTER } }
I20250905 19:59:56.851282 17987 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0809eccf0c14cfe89df1c01b7dbb575. Latest consensus state: current_term: 1 leader_uuid: "d0809eccf0c14cfe89df1c01b7dbb575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0809eccf0c14cfe89df1c01b7dbb575" member_type: VOTER } }
I20250905 19:59:56.851927 17986 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:56.852037 17987 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:56.869894 17990 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:56.876824 17990 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:56.883554 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:59:56.887912 17990 catalog_manager.cc:1349] Generated new cluster ID: f44c58ffeedb4dad96ed52155527f540
I20250905 19:59:56.888209 17990 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:56.907451 17990 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:59:56.908741 17990 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:56.924391 17990 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575: Generated new TSK 0
I20250905 19:59:56.925024 17990 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:56.951762 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:56.958976 18003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:56.960449 18004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:56.962148 18006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:56.962972 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:56.963781 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:56.963974 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:56.964138 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102396964119 us; error 0 us; skew 500 ppm
I20250905 19:59:56.964676 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:56.967446 15038 webserver.cc:480] Webserver started at http://127.14.175.129:45519/ using document root <none> and password file <none>
I20250905 19:59:56.967921 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:56.968099 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:56.968364 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:56.969457 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "01139077cb764b4f906a6a2982143393"
format_stamp: "Formatted at 2025-09-05 19:59:56 on dist-test-slave-22s0"
I20250905 19:59:56.974360 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 19:59:56.977792 18011 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:56.978662 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250905 19:59:56.978945 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "01139077cb764b4f906a6a2982143393"
format_stamp: "Formatted at 2025-09-05 19:59:56 on dist-test-slave-22s0"
I20250905 19:59:56.979252 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:57.004928 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:57.005926 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:57.017987 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:57.018589 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20250905 19:59:57.019040 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:57.019378 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:57.087010 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:40943
I20250905 19:59:57.087112 18081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:40943 every 8 connection(s)
I20250905 19:59:57.091846 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:59:57.091988 18082 heartbeater.cc:344] Connected to a master server at 127.14.175.190:45399
I20250905 19:59:57.092386 18082 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:57.093243 18082 heartbeater.cc:507] Master 127.14.175.190:45399 requested a full tablet report, sending...
I20250905 19:59:57.095481 17947 ts_manager.cc:194] Registered new tserver with Master: 01139077cb764b4f906a6a2982143393 (127.14.175.129:40943)
I20250905 19:59:57.097827 17947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34234
W20250905 19:59:57.098860 18084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:57.102301 18085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:57.103740 18087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:57.104811 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:57.105819 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:57.106083 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:57.106289 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102397106264 us; error 0 us; skew 500 ppm
I20250905 19:59:57.107000 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:57.109617 15038 webserver.cc:480] Webserver started at http://127.14.175.130:42587/ using document root <none> and password file <none>
I20250905 19:59:57.110246 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:57.110474 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:57.110821 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:57.112241 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-1-root/instance:
uuid: "c494ff57ff944ac5bebc30aeb6f6b4ab"
format_stamp: "Formatted at 2025-09-05 19:59:57 on dist-test-slave-22s0"
I20250905 19:59:57.117982 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20250905 19:59:57.122460 18092 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:57.123363 15038 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250905 19:59:57.123648 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-1-root
uuid: "c494ff57ff944ac5bebc30aeb6f6b4ab"
format_stamp: "Formatted at 2025-09-05 19:59:57 on dist-test-slave-22s0"
I20250905 19:59:57.123937 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:57.136283 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:57.137409 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:57.143630 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:57.143993 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250905 19:59:57.144253 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:57.144410 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:57.210047 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.130:37357
I20250905 19:59:57.210147 18161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.130:37357 every 8 connection(s)
I20250905 19:59:57.215190 18162 heartbeater.cc:344] Connected to a master server at 127.14.175.190:45399
I20250905 19:59:57.215538 18162 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:57.216315 18162 heartbeater.cc:507] Master 127.14.175.190:45399 requested a full tablet report, sending...
I20250905 19:59:57.218241 17947 ts_manager.cc:194] Registered new tserver with Master: c494ff57ff944ac5bebc30aeb6f6b4ab (127.14.175.130:37357)
I20250905 19:59:57.218734 15038 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.00515752s
I20250905 19:59:57.219637 17947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34258
I20250905 19:59:57.253237 18043 tablet_service.cc:1468] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20250905 19:59:57.254856 18043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:57.270704 18172 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: Bootstrap starting.
I20250905 19:59:57.275241 18172 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:57.279292 18172 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: No bootstrap required, opened a new log
I20250905 19:59:57.279655 18172 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20250905 19:59:57.281695 18172 raft_consensus.cc:357] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } }
I20250905 19:59:57.282097 18172 raft_consensus.cc:383] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:57.282328 18172 raft_consensus.cc:738] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01139077cb764b4f906a6a2982143393, State: Initialized, Role: FOLLOWER
I20250905 19:59:57.282908 18172 consensus_queue.cc:260] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } }
I20250905 19:59:57.283375 18172 raft_consensus.cc:397] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:57.283576 18172 raft_consensus.cc:491] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:57.283838 18172 raft_consensus.cc:3058] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:57.288305 18172 raft_consensus.cc:513] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } }
I20250905 19:59:57.288818 18172 leader_election.cc:304] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01139077cb764b4f906a6a2982143393; no voters:
I20250905 19:59:57.290151 18172 leader_election.cc:290] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:57.290469 18174 raft_consensus.cc:2802] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:57.291999 18082 heartbeater.cc:499] Master 127.14.175.190:45399 was elected leader, sending a full tablet report...
I20250905 19:59:57.293025 18172 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20250905 19:59:57.293174 18174 raft_consensus.cc:695] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 1 LEADER]: Becoming Leader. State: Replica: 01139077cb764b4f906a6a2982143393, State: Running, Role: LEADER
I20250905 19:59:57.293917 18174 consensus_queue.cc:237] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } }
I20250905 19:59:57.298286 18175 tablet_replica.cc:440] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01139077cb764b4f906a6a2982143393" committed_config { opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } } }
I20250905 19:59:57.298436 18176 tablet_replica.cc:440] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: TxnStatusTablet state changed. Reason: New leader 01139077cb764b4f906a6a2982143393. Latest consensus state: current_term: 1 leader_uuid: "01139077cb764b4f906a6a2982143393" committed_config { opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } } }
I20250905 19:59:57.298849 18175 tablet_replica.cc:443] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: This TxnStatusTablet replica's current role is: LEADER
W20250905 19:59:57.300462 17947 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250905 19:59:57.302991 18176 tablet_replica.cc:443] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: This TxnStatusTablet replica's current role is: LEADER
I20250905 19:59:57.305950 18178 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250905 19:59:57.306397 18178 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250905 19:59:57.310534 17947 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250905 19:59:57.361766 18183 ts_tablet_manager.cc:927] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: Initiating tablet copy from peer 01139077cb764b4f906a6a2982143393 (127.14.175.129:40943)
I20250905 19:59:57.363346 18183 tablet_copy_client.cc:323] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: tablet copy: Beginning tablet copy session from remote peer at address 127.14.175.129:40943
I20250905 19:59:57.371531 18067 tablet_copy_service.cc:140] P 01139077cb764b4f906a6a2982143393: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer c494ff57ff944ac5bebc30aeb6f6b4ab ({username='slave'} at 127.0.0.1:50614)
I20250905 19:59:57.372144 18067 tablet_copy_service.cc:161] P 01139077cb764b4f906a6a2982143393: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer c494ff57ff944ac5bebc30aeb6f6b4ab at {username='slave'} at 127.0.0.1:50614: session id = c494ff57ff944ac5bebc30aeb6f6b4ab-11111111111111111111111111111111
I20250905 19:59:57.379616 18067 tablet_copy_source_session.cc:215] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:57.382464 18183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:57.392843 18183 tablet_copy_client.cc:806] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:57.393396 18183 tablet_copy_client.cc:670] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:57.396986 18183 tablet_copy_client.cc:538] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:57.402849 18183 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: Bootstrap starting.
I20250905 19:59:57.424032 18183 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:57.424705 18183 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: Bootstrap complete.
I20250905 19:59:57.425402 18183 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.000s
I20250905 19:59:57.427376 18183 raft_consensus.cc:357] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } }
I20250905 19:59:57.427770 18183 raft_consensus.cc:738] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: c494ff57ff944ac5bebc30aeb6f6b4ab, State: Initialized, Role: NON_PARTICIPANT
I20250905 19:59:57.428341 18183 consensus_queue.cc:260] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } }
I20250905 19:59:57.430109 18186 tablet_replica.cc:440] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } } }
I20250905 19:59:57.430279 18162 heartbeater.cc:499] Master 127.14.175.190:45399 was elected leader, sending a full tablet report...
I20250905 19:59:57.430724 18186 tablet_replica.cc:443] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20250905 19:59:57.430969 18183 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20250905 19:59:57.432672 18067 tablet_copy_service.cc:342] P 01139077cb764b4f906a6a2982143393: Request end of tablet copy session c494ff57ff944ac5bebc30aeb6f6b4ab-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:50614
I20250905 19:59:57.433060 18067 tablet_copy_service.cc:434] P 01139077cb764b4f906a6a2982143393: ending tablet copy session c494ff57ff944ac5bebc30aeb6f6b4ab-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer c494ff57ff944ac5bebc30aeb6f6b4ab
W20250905 19:59:57.433293 17947 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
W20250905 19:59:57.436985 17947 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250905 19:59:57.458067 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
I20250905 19:59:57.480159 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:57.481192 15038 tablet_replica.cc:331] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: stopping tablet replica
I20250905 19:59:57.481667 15038 raft_consensus.cc:2241] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:57.482090 15038 raft_consensus.cc:2270] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:57.502847 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 19:59:57.517361 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:57.523962 18187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:57.528579 18188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:57.531797 18190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:57.532245 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:57.533450 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:57.533705 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:57.533900 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102397533883 us; error 0 us; skew 500 ppm
I20250905 19:59:57.534636 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:57.537353 15038 webserver.cc:480] Webserver started at http://127.14.175.129:45519/ using document root <none> and password file <none>
I20250905 19:59:57.537976 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:57.538203 15038 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250905 19:59:57.542569 15038 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20250905 19:59:57.545809 18195 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:57.546607 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250905 19:59:57.546929 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "01139077cb764b4f906a6a2982143393"
format_stamp: "Formatted at 2025-09-05 19:59:56 on dist-test-slave-22s0"
I20250905 19:59:57.547259 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:57.568552 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:57.569849 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:57.578048 18204 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 19:59:57.585381 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 19:59:57.585772 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20250905 19:59:57.586234 15038 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 19:59:57.591406 18204 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: Bootstrap starting.
I20250905 19:59:57.596534 15038 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 19:59:57.596859 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.005s sys 0.000s
I20250905 19:59:57.631722 18204 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:57.632685 18204 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: Bootstrap complete.
I20250905 19:59:57.633332 18204 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: Time spent bootstrapping tablet: real 0.042s user 0.023s sys 0.015s
I20250905 19:59:57.635984 18204 raft_consensus.cc:357] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } }
I20250905 19:59:57.636595 18204 raft_consensus.cc:738] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01139077cb764b4f906a6a2982143393, State: Initialized, Role: FOLLOWER
I20250905 19:59:57.637475 18204 consensus_queue.cc:260] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } }
I20250905 19:59:57.638190 18204 raft_consensus.cc:397] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:57.638567 18204 raft_consensus.cc:491] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:57.638932 18204 raft_consensus.cc:3058] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:57.645511 18204 raft_consensus.cc:513] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } }
I20250905 19:59:57.646152 18204 leader_election.cc:304] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01139077cb764b4f906a6a2982143393; no voters:
I20250905 19:59:57.647498 18204 leader_election.cc:290] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 19:59:57.647789 18229 raft_consensus.cc:2802] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:57.654928 18204 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393: Time spent starting tablet: real 0.021s user 0.014s sys 0.007s
I20250905 19:59:57.655733 18229 raft_consensus.cc:695] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 2 LEADER]: Becoming Leader. State: Replica: 01139077cb764b4f906a6a2982143393, State: Running, Role: LEADER
I20250905 19:59:57.656539 18229 consensus_queue.cc:237] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } }
I20250905 19:59:57.661216 18231 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: New leader 01139077cb764b4f906a6a2982143393. Latest consensus state: current_term: 2 leader_uuid: "01139077cb764b4f906a6a2982143393" committed_config { opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } } }
I20250905 19:59:57.661717 18231 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: LEADER
I20250905 19:59:57.671952 18230 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "01139077cb764b4f906a6a2982143393" committed_config { opid_index: -1 peers { permanent_uuid: "01139077cb764b4f906a6a2982143393" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 40943 } } }
I20250905 19:59:57.672618 18230 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: LEADER
I20250905 19:59:57.673794 18240 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250905 19:59:57.674286 18240 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250905 19:59:57.700708 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:40943
I20250905 19:59:57.700784 18272 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:40943 every 8 connection(s)
I20250905 19:59:57.707067 18273 heartbeater.cc:344] Connected to a master server at 127.14.175.190:45399
I20250905 19:59:57.707468 18273 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:57.708125 18273 heartbeater.cc:507] Master 127.14.175.190:45399 requested a full tablet report, sending...
I20250905 19:59:57.708932 15038 tablet_server.cc:178] TabletServer@127.14.175.129:40943 shutting down...
I20250905 19:59:57.711777 17947 ts_manager.cc:194] Re-registered known tserver with Master: 01139077cb764b4f906a6a2982143393 (127.14.175.129:40943)
W20250905 19:59:57.712474 17947 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250905 19:59:57.714134 17947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34292
I20250905 19:59:57.718353 18273 heartbeater.cc:499] Master 127.14.175.190:45399 was elected leader, sending a full tablet report...
W20250905 19:59:57.721385 17947 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250905 19:59:57.735612 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:57.736444 15038 tablet_replica.cc:331] stopping tablet replica
I20250905 19:59:57.736955 15038 raft_consensus.cc:2241] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:59:57.737401 15038 raft_consensus.cc:2270] T 11111111111111111111111111111111 P 01139077cb764b4f906a6a2982143393 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:57.758519 15038 tablet_server.cc:195] TabletServer@127.14.175.129:40943 shutdown complete.
I20250905 19:59:57.769304 15038 tablet_server.cc:178] TabletServer@127.14.175.130:0 shutting down...
I20250905 19:59:57.788275 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:57.789327 15038 tablet_replica.cc:331] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab: stopping tablet replica
I20250905 19:59:57.789806 15038 raft_consensus.cc:2241] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20250905 19:59:57.790184 15038 raft_consensus.cc:2270] T 11111111111111111111111111111111 P c494ff57ff944ac5bebc30aeb6f6b4ab [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20250905 19:59:57.800057 15038 tablet_server.cc:195] TabletServer@127.14.175.130:0 shutdown complete.
I20250905 19:59:57.810550 15038 master.cc:561] Master@127.14.175.190:45399 shutting down...
I20250905 19:59:57.831322 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:57.831844 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:57.832140 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P d0809eccf0c14cfe89df1c01b7dbb575: stopping tablet replica
I20250905 19:59:57.850838 15038 master.cc:583] Master@127.14.175.190:45399 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1164 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20250905 19:59:57.877250 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:44657
I20250905 19:59:57.878314 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:57.883666 18279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:57.883898 18280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:57.886912 18282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:57.887315 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:57.888038 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:57.888203 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:57.888360 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102397888346 us; error 0 us; skew 500 ppm
I20250905 19:59:57.888860 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:57.891650 15038 webserver.cc:480] Webserver started at http://127.14.175.190:33045/ using document root <none> and password file <none>
I20250905 19:59:57.892123 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:57.892290 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:57.892590 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:57.893723 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "f74d58e1812d4106b3b7819b5ca09bad"
format_stamp: "Formatted at 2025-09-05 19:59:57 on dist-test-slave-22s0"
I20250905 19:59:57.898343 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250905 19:59:57.901758 18287 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:57.902585 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 19:59:57.902853 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "f74d58e1812d4106b3b7819b5ca09bad"
format_stamp: "Formatted at 2025-09-05 19:59:57 on dist-test-slave-22s0"
I20250905 19:59:57.903124 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:57.924104 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:57.925088 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:57.974242 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:44657
I20250905 19:59:57.974329 18338 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:44657 every 8 connection(s)
I20250905 19:59:57.978533 18339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:57.988149 18339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad: Bootstrap starting.
I20250905 19:59:57.992365 18339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:57.996340 18339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad: No bootstrap required, opened a new log
I20250905 19:59:57.998189 18339 raft_consensus.cc:357] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74d58e1812d4106b3b7819b5ca09bad" member_type: VOTER }
I20250905 19:59:57.998593 18339 raft_consensus.cc:383] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:57.998802 18339 raft_consensus.cc:738] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74d58e1812d4106b3b7819b5ca09bad, State: Initialized, Role: FOLLOWER
I20250905 19:59:57.999368 18339 consensus_queue.cc:260] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74d58e1812d4106b3b7819b5ca09bad" member_type: VOTER }
I20250905 19:59:57.999801 18339 raft_consensus.cc:397] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:58.000011 18339 raft_consensus.cc:491] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:58.000257 18339 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:58.004660 18339 raft_consensus.cc:513] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74d58e1812d4106b3b7819b5ca09bad" member_type: VOTER }
I20250905 19:59:58.005154 18339 leader_election.cc:304] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f74d58e1812d4106b3b7819b5ca09bad; no voters:
I20250905 19:59:58.006433 18339 leader_election.cc:290] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:58.006778 18342 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:58.008549 18342 raft_consensus.cc:695] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [term 1 LEADER]: Becoming Leader. State: Replica: f74d58e1812d4106b3b7819b5ca09bad, State: Running, Role: LEADER
I20250905 19:59:58.009315 18342 consensus_queue.cc:237] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74d58e1812d4106b3b7819b5ca09bad" member_type: VOTER }
I20250905 19:59:58.009999 18339 sys_catalog.cc:564] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:58.012217 18343 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f74d58e1812d4106b3b7819b5ca09bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74d58e1812d4106b3b7819b5ca09bad" member_type: VOTER } }
I20250905 19:59:58.012321 18344 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [sys.catalog]: SysCatalogTable state changed. Reason: New leader f74d58e1812d4106b3b7819b5ca09bad. Latest consensus state: current_term: 1 leader_uuid: "f74d58e1812d4106b3b7819b5ca09bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74d58e1812d4106b3b7819b5ca09bad" member_type: VOTER } }
I20250905 19:59:58.012774 18343 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:58.012873 18344 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:58.015257 18347 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:58.024444 18347 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:58.038018 18347 catalog_manager.cc:1349] Generated new cluster ID: b445935e5f4a4f61adbe71fa738fa963
I20250905 19:59:58.038430 18347 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:58.042851 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:59:58.076666 18347 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:59:58.078346 18347 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:58.097198 18347 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad: Generated new TSK 0
I20250905 19:59:58.097892 18347 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:58.109042 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:58.115319 18360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:58.116871 18361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:58.118643 18363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:58.118969 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:58.119969 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:58.120230 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:58.120435 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102398120419 us; error 0 us; skew 500 ppm
I20250905 19:59:58.121133 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:58.127688 15038 webserver.cc:480] Webserver started at http://127.14.175.129:36709/ using document root <none> and password file <none>
I20250905 19:59:58.128321 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:58.128568 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:58.128896 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:58.130360 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "ad0da338919a4debbcaa5531908a6a5b"
format_stamp: "Formatted at 2025-09-05 19:59:58 on dist-test-slave-22s0"
I20250905 19:59:58.136375 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250905 19:59:58.140648 18368 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:58.141499 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:59:58.141775 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "ad0da338919a4debbcaa5531908a6a5b"
format_stamp: "Formatted at 2025-09-05 19:59:58 on dist-test-slave-22s0"
I20250905 19:59:58.142042 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:58.175570 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:58.176590 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:58.196056 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:58.196491 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250905 19:59:58.196835 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:58.197057 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:58.291718 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:42901
I20250905 19:59:58.291877 18438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:42901 every 8 connection(s)
I20250905 19:59:58.297340 18439 heartbeater.cc:344] Connected to a master server at 127.14.175.190:44657
I20250905 19:59:58.297679 18439 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:58.298334 18439 heartbeater.cc:507] Master 127.14.175.190:44657 requested a full tablet report, sending...
I20250905 19:59:58.300225 18304 ts_manager.cc:194] Registered new tserver with Master: ad0da338919a4debbcaa5531908a6a5b (127.14.175.129:42901)
I20250905 19:59:58.300791 15038 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004866593s
I20250905 19:59:58.301873 18304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56280
I20250905 19:59:58.335739 18404 tablet_service.cc:1468] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20250905 19:59:58.336953 18404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:58.348222 18449 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b: Bootstrap starting.
I20250905 19:59:58.352453 18449 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:58.356603 18449 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b: No bootstrap required, opened a new log
I20250905 19:59:58.356978 18449 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20250905 19:59:58.358853 18449 raft_consensus.cc:357] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "ad0da338919a4debbcaa5531908a6a5b" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42901 } }
I20250905 19:59:58.359251 18449 raft_consensus.cc:383] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:58.359465 18449 raft_consensus.cc:738] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad0da338919a4debbcaa5531908a6a5b, State: Initialized, Role: FOLLOWER
I20250905 19:59:58.359927 18449 consensus_queue.cc:260] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "ad0da338919a4debbcaa5531908a6a5b" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42901 } }
I20250905 19:59:58.360352 18449 raft_consensus.cc:397] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:58.360560 18449 raft_consensus.cc:491] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:58.360791 18449 raft_consensus.cc:3058] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:58.365453 18449 raft_consensus.cc:513] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "ad0da338919a4debbcaa5531908a6a5b" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42901 } }
I20250905 19:59:58.365939 18449 leader_election.cc:304] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad0da338919a4debbcaa5531908a6a5b; no voters:
I20250905 19:59:58.367302 18449 leader_election.cc:290] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:58.367635 18451 raft_consensus.cc:2802] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:58.369127 18439 heartbeater.cc:499] Master 127.14.175.190:44657 was elected leader, sending a full tablet report...
I20250905 19:59:58.370108 18449 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20250905 19:59:58.370291 18451 raft_consensus.cc:695] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [term 1 LEADER]: Becoming Leader. State: Replica: ad0da338919a4debbcaa5531908a6a5b, State: Running, Role: LEADER
I20250905 19:59:58.371060 18451 consensus_queue.cc:237] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "ad0da338919a4debbcaa5531908a6a5b" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42901 } }
I20250905 19:59:58.374107 18452 tablet_replica.cc:440] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad0da338919a4debbcaa5531908a6a5b" committed_config { opid_index: -1 peers { permanent_uuid: "ad0da338919a4debbcaa5531908a6a5b" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42901 } } }
I20250905 19:59:58.374650 18452 tablet_replica.cc:443] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b: This TxnStatusTablet replica's current role is: LEADER
W20250905 19:59:58.376579 18304 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250905 19:59:58.376392 18453 tablet_replica.cc:440] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b: TxnStatusTablet state changed. Reason: New leader ad0da338919a4debbcaa5531908a6a5b. Latest consensus state: current_term: 1 leader_uuid: "ad0da338919a4debbcaa5531908a6a5b" committed_config { opid_index: -1 peers { permanent_uuid: "ad0da338919a4debbcaa5531908a6a5b" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42901 } } }
I20250905 19:59:58.377071 18453 tablet_replica.cc:443] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b: This TxnStatusTablet replica's current role is: LEADER
I20250905 19:59:58.380654 18455 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250905 19:59:58.381108 18455 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250905 19:59:58.383417 18304 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
W20250905 19:59:58.430933 18391 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20250905 19:59:58.431221 18391 txn_status_manager.cc:244] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20250905 19:59:58.431540 18391 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20250905 19:59:58.439821 18391 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20250905 19:59:58.483927 18391 txn_status_manager.cc:244] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20250905 19:59:58.484313 18391 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20250905 19:59:58.492336 18391 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20250905 19:59:58.503408 18403 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:52912
W20250905 19:59:58.505627 18403 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:52912
W20250905 19:59:58.507678 18403 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:52912
W20250905 19:59:58.510375 18403 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:52912
W20250905 19:59:58.512480 18403 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:52912
W20250905 19:59:58.514638 18403 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:52912
W20250905 19:59:58.525173 18403 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:52922
W20250905 19:59:58.527137 18403 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:52922
W20250905 19:59:58.529026 18403 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:52922
W20250905 19:59:58.530853 18403 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:52922
W20250905 19:59:58.532716 18403 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:52922
W20250905 19:59:58.534561 18403 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:52922
I20250905 19:59:58.539775 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
I20250905 19:59:58.563345 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:58.564227 15038 tablet_replica.cc:331] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b: stopping tablet replica
I20250905 19:59:58.564913 15038 raft_consensus.cc:2241] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:58.565578 15038 raft_consensus.cc:2270] T 11111111111111111111111111111111 P ad0da338919a4debbcaa5531908a6a5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:58.589228 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 19:59:58.605432 15038 master.cc:561] Master@127.14.175.190:44657 shutting down...
I20250905 19:59:58.621840 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:58.622560 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:58.622938 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P f74d58e1812d4106b3b7819b5ca09bad: stopping tablet replica
I20250905 19:59:58.641412 15038 master.cc:583] Master@127.14.175.190:44657 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (787 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20250905 19:59:58.664913 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:36919
I20250905 19:59:58.665969 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:58.671130 18464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:58.672998 18465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:58.674204 18467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:58.674978 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:58.675778 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:58.675963 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:58.676121 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102398676094 us; error 0 us; skew 500 ppm
I20250905 19:59:58.676654 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:58.679553 15038 webserver.cc:480] Webserver started at http://127.14.175.190:45187/ using document root <none> and password file <none>
I20250905 19:59:58.680023 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:58.680218 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:58.680472 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:58.681547 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "9da99965526347dbaa60f7c3a638fb69"
format_stamp: "Formatted at 2025-09-05 19:59:58 on dist-test-slave-22s0"
I20250905 19:59:58.686127 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250905 19:59:58.689610 18472 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:58.690429 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 19:59:58.690766 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "9da99965526347dbaa60f7c3a638fb69"
format_stamp: "Formatted at 2025-09-05 19:59:58 on dist-test-slave-22s0"
I20250905 19:59:58.691111 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:58.722776 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:58.724068 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:58.766988 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:36919
I20250905 19:59:58.767058 18523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:36919 every 8 connection(s)
I20250905 19:59:58.771339 18524 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:58.781127 18524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69: Bootstrap starting.
I20250905 19:59:58.785214 18524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:58.788829 18524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69: No bootstrap required, opened a new log
I20250905 19:59:58.790781 18524 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da99965526347dbaa60f7c3a638fb69" member_type: VOTER }
I20250905 19:59:58.791158 18524 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:58.791375 18524 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da99965526347dbaa60f7c3a638fb69, State: Initialized, Role: FOLLOWER
I20250905 19:59:58.791859 18524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da99965526347dbaa60f7c3a638fb69" member_type: VOTER }
I20250905 19:59:58.792320 18524 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:58.792539 18524 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:58.792793 18524 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:58.797127 18524 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da99965526347dbaa60f7c3a638fb69" member_type: VOTER }
I20250905 19:59:58.797614 18524 leader_election.cc:304] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9da99965526347dbaa60f7c3a638fb69; no voters:
I20250905 19:59:58.798695 18524 leader_election.cc:290] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:58.798967 18527 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:58.800396 18527 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [term 1 LEADER]: Becoming Leader. State: Replica: 9da99965526347dbaa60f7c3a638fb69, State: Running, Role: LEADER
I20250905 19:59:58.801043 18527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da99965526347dbaa60f7c3a638fb69" member_type: VOTER }
I20250905 19:59:58.801656 18524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:58.806607 18528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9da99965526347dbaa60f7c3a638fb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da99965526347dbaa60f7c3a638fb69" member_type: VOTER } }
I20250905 19:59:58.806865 18529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9da99965526347dbaa60f7c3a638fb69. Latest consensus state: current_term: 1 leader_uuid: "9da99965526347dbaa60f7c3a638fb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da99965526347dbaa60f7c3a638fb69" member_type: VOTER } }
I20250905 19:59:58.807453 18528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:58.807543 18529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:58.809257 18535 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:58.813876 18535 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:58.816356 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:59:58.822423 18535 catalog_manager.cc:1349] Generated new cluster ID: 40d4a112a24f49ffbbea470d84227faa
I20250905 19:59:58.822712 18535 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:58.833817 18535 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:59:58.835064 18535 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:58.848634 18535 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69: Generated new TSK 0
I20250905 19:59:58.849205 18535 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:58.883634 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:58.889142 18545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:58.890266 18546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:58.891566 18548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:58.892021 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:58.892817 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:58.892998 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:58.893180 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102398893163 us; error 0 us; skew 500 ppm
I20250905 19:59:58.893723 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:58.895964 15038 webserver.cc:480] Webserver started at http://127.14.175.129:34063/ using document root <none> and password file <none>
I20250905 19:59:58.896417 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:58.896605 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:58.896850 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:58.897877 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "37257f340003480aabdfa88fb9026dde"
format_stamp: "Formatted at 2025-09-05 19:59:58 on dist-test-slave-22s0"
I20250905 19:59:58.902117 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250905 19:59:58.905395 18553 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:58.906080 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:59:58.906351 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "37257f340003480aabdfa88fb9026dde"
format_stamp: "Formatted at 2025-09-05 19:59:58 on dist-test-slave-22s0"
I20250905 19:59:58.906664 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:58.919059 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:58.920022 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:58.925890 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:58.926163 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:58.926402 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:58.926585 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250905 19:59:58.974702 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:37325
I20250905 19:59:58.974803 18623 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:37325 every 8 connection(s)
I20250905 19:59:58.979265 18624 heartbeater.cc:344] Connected to a master server at 127.14.175.190:36919
I20250905 19:59:58.979591 18624 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:58.980273 18624 heartbeater.cc:507] Master 127.14.175.190:36919 requested a full tablet report, sending...
I20250905 19:59:58.981985 18489 ts_manager.cc:194] Registered new tserver with Master: 37257f340003480aabdfa88fb9026dde (127.14.175.129:37325)
I20250905 19:59:58.982132 15038 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.003788163s
I20250905 19:59:58.983651 18489 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37266
I20250905 19:59:59.011287 18589 tablet_service.cc:1468] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20250905 19:59:59.012363 18589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:59.023092 18634 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde: Bootstrap starting.
I20250905 19:59:59.027266 18634 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:59.030912 18634 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde: No bootstrap required, opened a new log
I20250905 19:59:59.031278 18634 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20250905 19:59:59.033077 18634 raft_consensus.cc:357] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "37257f340003480aabdfa88fb9026dde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37325 } }
I20250905 19:59:59.033492 18634 raft_consensus.cc:383] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:59.033713 18634 raft_consensus.cc:738] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37257f340003480aabdfa88fb9026dde, State: Initialized, Role: FOLLOWER
I20250905 19:59:59.034204 18634 consensus_queue.cc:260] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "37257f340003480aabdfa88fb9026dde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37325 } }
I20250905 19:59:59.034693 18634 raft_consensus.cc:397] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:59.034901 18634 raft_consensus.cc:491] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:59.035154 18634 raft_consensus.cc:3058] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:59.039785 18634 raft_consensus.cc:513] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "37257f340003480aabdfa88fb9026dde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37325 } }
I20250905 19:59:59.040292 18634 leader_election.cc:304] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37257f340003480aabdfa88fb9026dde; no voters:
I20250905 19:59:59.041355 18634 leader_election.cc:290] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:59.041689 18636 raft_consensus.cc:2802] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:59.042995 18636 raft_consensus.cc:695] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [term 1 LEADER]: Becoming Leader. State: Replica: 37257f340003480aabdfa88fb9026dde, State: Running, Role: LEADER
I20250905 19:59:59.043171 18624 heartbeater.cc:499] Master 127.14.175.190:36919 was elected leader, sending a full tablet report...
I20250905 19:59:59.043707 18636 consensus_queue.cc:237] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "37257f340003480aabdfa88fb9026dde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37325 } }
I20250905 19:59:59.044287 18634 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20250905 19:59:59.047206 18638 tablet_replica.cc:440] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde: TxnStatusTablet state changed. Reason: New leader 37257f340003480aabdfa88fb9026dde. Latest consensus state: current_term: 1 leader_uuid: "37257f340003480aabdfa88fb9026dde" committed_config { opid_index: -1 peers { permanent_uuid: "37257f340003480aabdfa88fb9026dde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37325 } } }
I20250905 19:59:59.047719 18638 tablet_replica.cc:443] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde: This TxnStatusTablet replica's current role is: LEADER
I20250905 19:59:59.049728 18637 tablet_replica.cc:440] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37257f340003480aabdfa88fb9026dde" committed_config { opid_index: -1 peers { permanent_uuid: "37257f340003480aabdfa88fb9026dde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37325 } } }
I20250905 19:59:59.050465 18637 tablet_replica.cc:443] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde: This TxnStatusTablet replica's current role is: LEADER
I20250905 19:59:59.050769 18641 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
W20250905 19:59:59.050999 18489 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250905 19:59:59.051250 18641 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250905 19:59:59.054913 18489 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250905 19:59:59.068737 18589 tablet_service.cc:1468] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20250905 19:59:59.069720 18589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:59.079598 18634 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde: Bootstrap starting.
I20250905 19:59:59.083662 18634 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:59.087352 18634 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde: No bootstrap required, opened a new log
I20250905 19:59:59.087700 18634 ts_tablet_manager.cc:1397] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20250905 19:59:59.089478 18634 raft_consensus.cc:357] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "37257f340003480aabdfa88fb9026dde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37325 } }
I20250905 19:59:59.089864 18634 raft_consensus.cc:383] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:59.090075 18634 raft_consensus.cc:738] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37257f340003480aabdfa88fb9026dde, State: Initialized, Role: FOLLOWER
I20250905 19:59:59.090600 18634 consensus_queue.cc:260] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "37257f340003480aabdfa88fb9026dde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37325 } }
I20250905 19:59:59.091030 18634 raft_consensus.cc:397] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:59.091253 18634 raft_consensus.cc:491] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:59.091496 18634 raft_consensus.cc:3058] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:59.096067 18634 raft_consensus.cc:513] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "37257f340003480aabdfa88fb9026dde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37325 } }
I20250905 19:59:59.096582 18634 leader_election.cc:304] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37257f340003480aabdfa88fb9026dde; no voters:
I20250905 19:59:59.097018 18634 leader_election.cc:290] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:59.097182 18637 raft_consensus.cc:2802] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:59.097690 18637 raft_consensus.cc:695] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [term 1 LEADER]: Becoming Leader. State: Replica: 37257f340003480aabdfa88fb9026dde, State: Running, Role: LEADER
I20250905 19:59:59.098423 18634 ts_tablet_manager.cc:1428] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde: Time spent starting tablet: real 0.010s user 0.002s sys 0.006s
I20250905 19:59:59.098305 18637 consensus_queue.cc:237] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "37257f340003480aabdfa88fb9026dde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37325 } }
W20250905 19:59:59.103497 18489 catalog_manager.cc:5415] Ignoring report from unknown tablet 22222222222222222222222222222222
W20250905 19:59:59.107033 18489 catalog_manager.cc:5415] Ignoring report from unknown tablet 22222222222222222222222222222222
I20250905 19:59:59.127941 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
I20250905 19:59:59.147166 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:59:59.147898 15038 tablet_replica.cc:331] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde: stopping tablet replica
I20250905 19:59:59.148486 15038 raft_consensus.cc:2241] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:59.148921 15038 raft_consensus.cc:2270] T 22222222222222222222222222222222 P 37257f340003480aabdfa88fb9026dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:59.151340 15038 tablet_replica.cc:331] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde: stopping tablet replica
I20250905 19:59:59.151778 15038 raft_consensus.cc:2241] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:59.152168 15038 raft_consensus.cc:2270] T 11111111111111111111111111111111 P 37257f340003480aabdfa88fb9026dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:59.163197 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 19:59:59.174633 15038 master.cc:561] Master@127.14.175.190:36919 shutting down...
I20250905 19:59:59.188351 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:59:59.188956 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:59.189417 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P 9da99965526347dbaa60f7c3a638fb69: stopping tablet replica
I20250905 19:59:59.208813 15038 master.cc:583] Master@127.14.175.190:36919 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (568 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (3341 ms total)
[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/0
I20250905 19:59:59.233095 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:38333
I20250905 19:59:59.234164 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:59.238708 18650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:59.239874 18651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:59.241307 18653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:59.242036 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:59.242787 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:59.242966 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:59.243119 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102399243100 us; error 0 us; skew 500 ppm
I20250905 19:59:59.243628 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:59.245752 15038 webserver.cc:480] Webserver started at http://127.14.175.190:44155/ using document root <none> and password file <none>
I20250905 19:59:59.246212 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:59.246395 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:59.246688 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:59.247735 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "70fe8885f0a948abbdbb57d4c27093fc"
format_stamp: "Formatted at 2025-09-05 19:59:59 on dist-test-slave-22s0"
I20250905 19:59:59.251962 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 19:59:59.255064 18658 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:59.255717 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 19:59:59.255971 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "70fe8885f0a948abbdbb57d4c27093fc"
format_stamp: "Formatted at 2025-09-05 19:59:59 on dist-test-slave-22s0"
I20250905 19:59:59.256227 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:59.274616 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:59.275592 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:59.309212 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:38333
I20250905 19:59:59.309302 18709 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:38333 every 8 connection(s)
I20250905 19:59:59.312897 18710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:59.322927 18710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc: Bootstrap starting.
I20250905 19:59:59.327124 18710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:59.330879 18710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc: No bootstrap required, opened a new log
I20250905 19:59:59.332682 18710 raft_consensus.cc:357] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fe8885f0a948abbdbb57d4c27093fc" member_type: VOTER }
I20250905 19:59:59.333029 18710 raft_consensus.cc:383] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:59.333200 18710 raft_consensus.cc:738] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70fe8885f0a948abbdbb57d4c27093fc, State: Initialized, Role: FOLLOWER
I20250905 19:59:59.333715 18710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fe8885f0a948abbdbb57d4c27093fc" member_type: VOTER }
I20250905 19:59:59.334136 18710 raft_consensus.cc:397] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:59:59.334321 18710 raft_consensus.cc:491] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:59:59.334569 18710 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:59.338801 18710 raft_consensus.cc:513] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fe8885f0a948abbdbb57d4c27093fc" member_type: VOTER }
I20250905 19:59:59.339229 18710 leader_election.cc:304] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70fe8885f0a948abbdbb57d4c27093fc; no voters:
I20250905 19:59:59.340240 18710 leader_election.cc:290] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:59:59.340574 18713 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:59.341843 18713 raft_consensus.cc:695] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [term 1 LEADER]: Becoming Leader. State: Replica: 70fe8885f0a948abbdbb57d4c27093fc, State: Running, Role: LEADER
I20250905 19:59:59.342481 18713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fe8885f0a948abbdbb57d4c27093fc" member_type: VOTER }
I20250905 19:59:59.343168 18710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:59:59.345844 18715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70fe8885f0a948abbdbb57d4c27093fc. Latest consensus state: current_term: 1 leader_uuid: "70fe8885f0a948abbdbb57d4c27093fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fe8885f0a948abbdbb57d4c27093fc" member_type: VOTER } }
I20250905 19:59:59.346474 18715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:59.345739 18714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70fe8885f0a948abbdbb57d4c27093fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fe8885f0a948abbdbb57d4c27093fc" member_type: VOTER } }
I20250905 19:59:59.347348 18714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [sys.catalog]: This master's current role is: LEADER
I20250905 19:59:59.351230 18719 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:59:59.356444 18719 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:59:59.357465 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:59:59.364719 18719 catalog_manager.cc:1349] Generated new cluster ID: e615f8f1ad374ad782d1aa2e2e1abef9
I20250905 19:59:59.364954 18719 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:59:59.388121 18719 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:59:59.389325 18719 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:59:59.402792 18719 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc: Generated new TSK 0
I20250905 19:59:59.403324 18719 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:59:59.423904 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:59.429772 18731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:59.431195 18732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:59.433640 18734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:59.434185 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:59.437467 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:59.437669 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:59.437829 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102399437810 us; error 0 us; skew 500 ppm
I20250905 19:59:59.438323 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:59.440593 15038 webserver.cc:480] Webserver started at http://127.14.175.129:40013/ using document root <none> and password file <none>
I20250905 19:59:59.441069 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:59.441262 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:59.441516 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:59.442613 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "9f2bc9ce641f42c1b9149e584c915bde"
format_stamp: "Formatted at 2025-09-05 19:59:59 on dist-test-slave-22s0"
I20250905 19:59:59.446951 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20250905 19:59:59.450083 18739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:59.450780 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 19:59:59.451045 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "9f2bc9ce641f42c1b9149e584c915bde"
format_stamp: "Formatted at 2025-09-05 19:59:59 on dist-test-slave-22s0"
I20250905 19:59:59.451294 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:59.466207 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:59.467209 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:59.468468 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:59.470569 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:59.470752 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:59.470970 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:59.471114 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:59.506853 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:37751
I20250905 19:59:59.506963 18801 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:37751 every 8 connection(s)
I20250905 19:59:59.511349 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:59.518862 18806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:59.520224 18807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:59.523782 15038 server_base.cc:1047] running on GCE node
W20250905 19:59:59.523869 18809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:59.524924 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:59:59.525128 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:59.525296 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102399525280 us; error 0 us; skew 500 ppm
I20250905 19:59:59.525410 18802 heartbeater.cc:344] Connected to a master server at 127.14.175.190:38333
I20250905 19:59:59.525861 18802 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:59.526098 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:59.526706 18802 heartbeater.cc:507] Master 127.14.175.190:38333 requested a full tablet report, sending...
I20250905 19:59:59.528716 15038 webserver.cc:480] Webserver started at http://127.14.175.130:42703/ using document root <none> and password file <none>
I20250905 19:59:59.529037 18675 ts_manager.cc:194] Registered new tserver with Master: 9f2bc9ce641f42c1b9149e584c915bde (127.14.175.129:37751)
I20250905 19:59:59.529359 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:59.529636 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:59.529950 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:59.531350 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-1-root/instance:
uuid: "33c62532982e4fa38fae8e3991477a2d"
format_stamp: "Formatted at 2025-09-05 19:59:59 on dist-test-slave-22s0"
I20250905 19:59:59.531404 18675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35654
I20250905 19:59:59.536365 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20250905 19:59:59.539391 18814 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:59.540066 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 19:59:59.540336 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-1-root
uuid: "33c62532982e4fa38fae8e3991477a2d"
format_stamp: "Formatted at 2025-09-05 19:59:59 on dist-test-slave-22s0"
I20250905 19:59:59.540588 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:59.555053 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:59.556020 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:59.557324 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:59.559451 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:59.559617 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:59.559854 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:59.560009 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:59.596349 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.130:39041
I20250905 19:59:59.596464 18876 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.130:39041 every 8 connection(s)
I20250905 19:59:59.600764 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:59:59.607066 18880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:59.608422 18881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:59:59.610363 18883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:59:59.611235 15038 server_base.cc:1047] running on GCE node
I20250905 19:59:59.612192 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250905 19:59:59.612305 18877 heartbeater.cc:344] Connected to a master server at 127.14.175.190:38333
W20250905 19:59:59.612457 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:59:59.612707 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102399612689 us; error 0 us; skew 500 ppm
I20250905 19:59:59.612739 18877 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:59.613294 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:59:59.613466 18877 heartbeater.cc:507] Master 127.14.175.190:38333 requested a full tablet report, sending...
I20250905 19:59:59.615483 18675 ts_manager.cc:194] Registered new tserver with Master: 33c62532982e4fa38fae8e3991477a2d (127.14.175.130:39041)
I20250905 19:59:59.615824 15038 webserver.cc:480] Webserver started at http://127.14.175.131:38143/ using document root <none> and password file <none>
I20250905 19:59:59.616290 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 19:59:59.616465 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:59:59.616696 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:59:59.616907 18675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35664
I20250905 19:59:59.617729 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-2-root/instance:
uuid: "a05ee81c2af44b0ba999a3d8df7bbf97"
format_stamp: "Formatted at 2025-09-05 19:59:59 on dist-test-slave-22s0"
I20250905 19:59:59.621876 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 19:59:59.624913 18888 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:59.625658 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:59:59.626012 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-2-root
uuid: "a05ee81c2af44b0ba999a3d8df7bbf97"
format_stamp: "Formatted at 2025-09-05 19:59:59 on dist-test-slave-22s0"
I20250905 19:59:59.626291 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:59:59.643507 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:59:59.644490 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:59:59.645903 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:59:59.648026 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:59:59.648214 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:59.648468 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:59:59.648624 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:59:59.684600 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.131:35135
I20250905 19:59:59.684705 18950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.131:35135 every 8 connection(s)
I20250905 19:59:59.696902 18951 heartbeater.cc:344] Connected to a master server at 127.14.175.190:38333
I20250905 19:59:59.697274 18951 heartbeater.cc:461] Registering TS with master...
I20250905 19:59:59.697937 18951 heartbeater.cc:507] Master 127.14.175.190:38333 requested a full tablet report, sending...
I20250905 19:59:59.699560 18675 ts_manager.cc:194] Registered new tserver with Master: a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131:35135)
I20250905 19:59:59.699776 15038 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012207288s
I20250905 19:59:59.700788 18675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35672
I20250905 19:59:59.718204 15038 test_util.cc:276] Using random seed: 1214153511
I20250905 19:59:59.738610 18675 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35688:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:59:59.775418 18767 tablet_service.cc:1468] Processing CreateTablet for tablet aaac0518535247ce87fd064059f1df20 (DEFAULT_TABLE table=test-workload [id=582473e3ba3b46f08dba63f6284df818]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:59.776538 18767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaac0518535247ce87fd064059f1df20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:59.777669 18842 tablet_service.cc:1468] Processing CreateTablet for tablet aaac0518535247ce87fd064059f1df20 (DEFAULT_TABLE table=test-workload [id=582473e3ba3b46f08dba63f6284df818]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:59.778966 18842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaac0518535247ce87fd064059f1df20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:59.786093 18916 tablet_service.cc:1468] Processing CreateTablet for tablet aaac0518535247ce87fd064059f1df20 (DEFAULT_TABLE table=test-workload [id=582473e3ba3b46f08dba63f6284df818]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:59:59.787353 18916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaac0518535247ce87fd064059f1df20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:59.792330 18971 tablet_bootstrap.cc:492] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde: Bootstrap starting.
I20250905 19:59:59.795567 18972 tablet_bootstrap.cc:492] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d: Bootstrap starting.
I20250905 19:59:59.798269 18971 tablet_bootstrap.cc:654] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:59.802147 18972 tablet_bootstrap.cc:654] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:59.802304 18973 tablet_bootstrap.cc:492] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Bootstrap starting.
I20250905 19:59:59.804569 18971 tablet_bootstrap.cc:492] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde: No bootstrap required, opened a new log
I20250905 19:59:59.804961 18971 ts_tablet_manager.cc:1397] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20250905 19:59:59.807467 18972 tablet_bootstrap.cc:492] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d: No bootstrap required, opened a new log
I20250905 19:59:59.807367 18971 raft_consensus.cc:357] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } }
I20250905 19:59:59.808024 18972 ts_tablet_manager.cc:1397] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250905 19:59:59.808166 18971 raft_consensus.cc:383] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:59.808691 18971 raft_consensus.cc:738] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2bc9ce641f42c1b9149e584c915bde, State: Initialized, Role: FOLLOWER
I20250905 19:59:59.809131 18973 tablet_bootstrap.cc:654] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Neither blocks nor log segments found. Creating new log.
I20250905 19:59:59.809437 18971 consensus_queue.cc:260] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } }
I20250905 19:59:59.810995 18972 raft_consensus.cc:357] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } }
I20250905 19:59:59.811712 18972 raft_consensus.cc:383] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:59.811748 18802 heartbeater.cc:499] Master 127.14.175.190:38333 was elected leader, sending a full tablet report...
I20250905 19:59:59.812234 18971 ts_tablet_manager.cc:1428] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20250905 19:59:59.812700 18972 raft_consensus.cc:738] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c62532982e4fa38fae8e3991477a2d, State: Initialized, Role: FOLLOWER
I20250905 19:59:59.814134 18972 consensus_queue.cc:260] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } }
I20250905 19:59:59.814683 18973 tablet_bootstrap.cc:492] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: No bootstrap required, opened a new log
I20250905 19:59:59.815200 18973 ts_tablet_manager.cc:1397] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.002s
I20250905 19:59:59.816569 18877 heartbeater.cc:499] Master 127.14.175.190:38333 was elected leader, sending a full tablet report...
I20250905 19:59:59.818022 18973 raft_consensus.cc:357] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } }
I20250905 19:59:59.818733 18973 raft_consensus.cc:383] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:59:59.819012 18973 raft_consensus.cc:738] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05ee81c2af44b0ba999a3d8df7bbf97, State: Initialized, Role: FOLLOWER
I20250905 19:59:59.819132 18972 ts_tablet_manager.cc:1428] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250905 19:59:59.819869 18973 consensus_queue.cc:260] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } }
I20250905 19:59:59.822606 18973 ts_tablet_manager.cc:1428] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20250905 19:59:59.823274 18951 heartbeater.cc:499] Master 127.14.175.190:38333 was elected leader, sending a full tablet report...
I20250905 19:59:59.965127 18976 raft_consensus.cc:491] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:59.965618 18976 raft_consensus.cc:513] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } }
I20250905 19:59:59.967617 18976 leader_election.cc:290] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33c62532982e4fa38fae8e3991477a2d (127.14.175.130:39041), a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131:35135)
I20250905 19:59:59.976346 18852 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aaac0518535247ce87fd064059f1df20" candidate_uuid: "9f2bc9ce641f42c1b9149e584c915bde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c62532982e4fa38fae8e3991477a2d" is_pre_election: true
I20250905 19:59:59.976938 18852 raft_consensus.cc:2466] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f2bc9ce641f42c1b9149e584c915bde in term 0.
I20250905 19:59:59.977217 18926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aaac0518535247ce87fd064059f1df20" candidate_uuid: "9f2bc9ce641f42c1b9149e584c915bde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" is_pre_election: true
I20250905 19:59:59.977883 18926 raft_consensus.cc:2466] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f2bc9ce641f42c1b9149e584c915bde in term 0.
I20250905 19:59:59.978055 18742 leader_election.cc:304] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c62532982e4fa38fae8e3991477a2d, 9f2bc9ce641f42c1b9149e584c915bde; no voters:
I20250905 19:59:59.978914 18976 raft_consensus.cc:2802] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:59:59.979278 18976 raft_consensus.cc:491] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:59.979559 18976 raft_consensus.cc:3058] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:59.984179 18976 raft_consensus.cc:513] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } }
I20250905 19:59:59.985525 18976 leader_election.cc:290] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [CANDIDATE]: Term 1 election: Requested vote from peers 33c62532982e4fa38fae8e3991477a2d (127.14.175.130:39041), a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131:35135)
I20250905 19:59:59.986318 18852 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aaac0518535247ce87fd064059f1df20" candidate_uuid: "9f2bc9ce641f42c1b9149e584c915bde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c62532982e4fa38fae8e3991477a2d"
I20250905 19:59:59.986536 18926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aaac0518535247ce87fd064059f1df20" candidate_uuid: "9f2bc9ce641f42c1b9149e584c915bde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97"
I20250905 19:59:59.986802 18852 raft_consensus.cc:3058] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:59.987040 18926 raft_consensus.cc:3058] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:59:59.991503 18926 raft_consensus.cc:2466] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f2bc9ce641f42c1b9149e584c915bde in term 1.
I20250905 19:59:59.991526 18852 raft_consensus.cc:2466] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f2bc9ce641f42c1b9149e584c915bde in term 1.
I20250905 19:59:59.992574 18742 leader_election.cc:304] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f2bc9ce641f42c1b9149e584c915bde, a05ee81c2af44b0ba999a3d8df7bbf97; no voters:
I20250905 19:59:59.993165 18976 raft_consensus.cc:2802] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:59:59.993937 18976 raft_consensus.cc:695] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 1 LEADER]: Becoming Leader. State: Replica: 9f2bc9ce641f42c1b9149e584c915bde, State: Running, Role: LEADER
I20250905 19:59:59.994637 18976 consensus_queue.cc:237] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } }
I20250905 20:00:00.001945 18674 catalog_manager.cc:5582] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f2bc9ce641f42c1b9149e584c915bde (127.14.175.129). New cstate: current_term: 1 leader_uuid: "9f2bc9ce641f42c1b9149e584c915bde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } } }
I20250905 20:00:00.254052 18842 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:00.257347 18767 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 20:00:00.257541 18916 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:00.448911 18976 consensus_queue.cc:1035] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
70fe8885f0a948abbdbb57d4c27093fc | 127.14.175.190:38333 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
33c62532982e4fa38fae8e3991477a2d | 127.14.175.130:39041 | HEALTHY | <none> | 0 | 0
9f2bc9ce641f42c1b9149e584c915bde | 127.14.175.129:37751 | HEALTHY | <none> | 1 | 0
a05ee81c2af44b0ba999a3d8df7bbf97 | 127.14.175.131:35135 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
I20250905 20:00:00.466439 18976 consensus_queue.cc:1035] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [LEADER]: Connected to new peer: Peer: permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet aaac0518535247ce87fd064059f1df20 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
9f2bc9ce641f42c1b9149e584c915bde (127.14.175.129:37751): RUNNING [LEADER]
33c62532982e4fa38fae8e3991477a2d (127.14.175.130:39041): RUNNING
a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131:35135): RUNNING
All reported replicas are:
A = 9f2bc9ce641f42c1b9149e584c915bde
B = 33c62532982e4fa38fae8e3991477a2d
C = a05ee81c2af44b0ba999a3d8df7bbf97
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250905 20:00:00.477877 15038 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250905 20:00:00.830281 18842 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:00.833756 18767 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 20:00:00.837497 18916 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
70fe8885f0a948abbdbb57d4c27093fc | 127.14.175.190:38333 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
33c62532982e4fa38fae8e3991477a2d | 127.14.175.130:39041 | HEALTHY | <none> | 0 | 0
9f2bc9ce641f42c1b9149e584c915bde | 127.14.175.129:37751 | HEALTHY | <none> | 1 | 0
a05ee81c2af44b0ba999a3d8df7bbf97 | 127.14.175.131:35135 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250905 20:00:01.027253 15038 log_verifier.cc:126] Checking tablet aaac0518535247ce87fd064059f1df20
I20250905 20:00:01.085242 15038 log_verifier.cc:177] Verified matching terms for 1 ops in tablet aaac0518535247ce87fd064059f1df20
I20250905 20:00:01.085850 15038 tablet_replica.cc:331] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: stopping tablet replica
I20250905 20:00:01.086369 15038 raft_consensus.cc:2241] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 20:00:01.086874 15038 raft_consensus.cc:2270] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 20:00:01.280798 18842 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:01.283826 18767 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 20:00:01.288187 18916 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250905 20:00:01.349901 18742 consensus_peers.cc:489] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde -> Peer a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131:35135): Couldn't send request to peer a05ee81c2af44b0ba999a3d8df7bbf97. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20250905 20:00:01.371206 18976 raft_consensus.cc:1010] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde: Attempting to remove follower a05ee81c2af44b0ba999a3d8df7bbf97 from the Raft config. Reason: The tablet replica hosted on peer a05ee81c2af44b0ba999a3d8df7bbf97 has failed
I20250905 20:00:01.373606 18976 consensus_queue.cc:237] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } }
I20250905 20:00:01.379534 18852 raft_consensus.cc:1273] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [term 1 FOLLOWER]: Refusing update from remote peer 9f2bc9ce641f42c1b9149e584c915bde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 20:00:01.381017 18976 consensus_queue.cc:1035] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 20:00:01.393405 18976 raft_consensus.cc:2953] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } }
I20250905 20:00:01.396924 18852 raft_consensus.cc:2953] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } }
I20250905 20:00:01.409986 18675 catalog_manager.cc:5582] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde reported cstate change: config changed from index -1 to 2, VOTER a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131) evicted. New cstate: current_term: 1 leader_uuid: "9f2bc9ce641f42c1b9149e584c915bde" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } }
I20250905 20:00:01.438220 18916 tablet_service.cc:1515] Processing DeleteTablet for tablet aaac0518535247ce87fd064059f1df20 with delete_type TABLET_DATA_TOMBSTONED (TS a05ee81c2af44b0ba999a3d8df7bbf97 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:58784
I20250905 20:00:01.444109 19073 ts_tablet_manager.cc:1905] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 20:00:01.444567 18777 consensus_queue.cc:237] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } }
W20250905 20:00:01.451352 18742 consensus_peers.cc:489] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde -> Peer a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131:35135): Couldn't send request to peer a05ee81c2af44b0ba999a3d8df7bbf97. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20250905 20:00:01.454123 18852 raft_consensus.cc:1273] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [term 1 FOLLOWER]: Refusing update from remote peer 9f2bc9ce641f42c1b9149e584c915bde: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 20:00:01.455783 18976 consensus_queue.cc:1035] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 20:00:01.460397 19073 ts_tablet_manager.cc:1918] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250905 20:00:01.460887 19073 log.cc:1199] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Deleting WAL directory at /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1757102344413625-15038-0/minicluster-data/ts-2-root/wals/aaac0518535247ce87fd064059f1df20
I20250905 20:00:01.462710 18661 catalog_manager.cc:4928] TS a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131:35135): tablet aaac0518535247ce87fd064059f1df20 (table test-workload [id=582473e3ba3b46f08dba63f6284df818]) successfully deleted
I20250905 20:00:01.464279 19066 raft_consensus.cc:2953] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } } }
I20250905 20:00:01.472580 18852 raft_consensus.cc:2953] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } } }
I20250905 20:00:01.475725 18660 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:VOTER RPC for tablet aaac0518535247ce87fd064059f1df20 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20250905 20:00:01.479305 18675 catalog_manager.cc:5582] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde reported cstate change: config changed from index 2 to 3, VOTER a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131) added. New cstate: current_term: 1 leader_uuid: "9f2bc9ce641f42c1b9149e584c915bde" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
70fe8885f0a948abbdbb57d4c27093fc | 127.14.175.190:38333 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
33c62532982e4fa38fae8e3991477a2d | 127.14.175.130:39041 | HEALTHY | <none> | 0 | 0
9f2bc9ce641f42c1b9149e584c915bde | 127.14.175.129:37751 | HEALTHY | <none> | 1 | 0
a05ee81c2af44b0ba999a3d8df7bbf97 | 127.14.175.131:35135 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet aaac0518535247ce87fd064059f1df20 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
9f2bc9ce641f42c1b9149e584c915bde (127.14.175.129:37751): RUNNING [LEADER]
33c62532982e4fa38fae8e3991477a2d (127.14.175.130:39041): RUNNING
a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131:35135): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = 9f2bc9ce641f42c1b9149e584c915bde
B = 33c62532982e4fa38fae8e3991477a2d
C = a05ee81c2af44b0ba999a3d8df7bbf97
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250905 20:00:01.516956 15038 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250905 20:00:01.871752 19100 ts_tablet_manager.cc:927] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Initiating tablet copy from peer 9f2bc9ce641f42c1b9149e584c915bde (127.14.175.129:37751)
I20250905 20:00:01.878643 18767 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 20:00:01.880512 19100 tablet_copy_client.cc:323] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: tablet copy: Beginning tablet copy session from remote peer at address 127.14.175.129:37751
I20250905 20:00:01.880774 18842 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:01.891371 18916 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:01.894141 18787 tablet_copy_service.cc:140] P 9f2bc9ce641f42c1b9149e584c915bde: Received BeginTabletCopySession request for tablet aaac0518535247ce87fd064059f1df20 from peer a05ee81c2af44b0ba999a3d8df7bbf97 ({username='slave'} at 127.0.0.1:47024)
I20250905 20:00:01.894650 18787 tablet_copy_service.cc:161] P 9f2bc9ce641f42c1b9149e584c915bde: Beginning new tablet copy session on tablet aaac0518535247ce87fd064059f1df20 from peer a05ee81c2af44b0ba999a3d8df7bbf97 at {username='slave'} at 127.0.0.1:47024: session id = a05ee81c2af44b0ba999a3d8df7bbf97-aaac0518535247ce87fd064059f1df20
I20250905 20:00:01.900369 18787 tablet_copy_source_session.cc:215] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 20:00:01.902921 19100 ts_tablet_manager.cc:1905] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250905 20:00:01.920104 19100 ts_tablet_manager.cc:1918] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20250905 20:00:01.920919 19100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaac0518535247ce87fd064059f1df20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 20:00:01.928508 19100 tablet_copy_client.cc:806] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: tablet copy: Starting download of 0 data blocks...
I20250905 20:00:01.929092 19100 tablet_copy_client.cc:670] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: tablet copy: Starting download of 1 WAL segments...
I20250905 20:00:01.933161 19100 tablet_copy_client.cc:538] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 20:00:01.942404 19100 tablet_bootstrap.cc:492] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Bootstrap starting.
I20250905 20:00:01.975803 19100 tablet_bootstrap.cc:492] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 20:00:01.976929 19100 tablet_bootstrap.cc:492] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Bootstrap complete.
I20250905 20:00:01.977689 19100 ts_tablet_manager.cc:1397] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Time spent bootstrapping tablet: real 0.035s user 0.032s sys 0.005s
I20250905 20:00:01.980343 19100 raft_consensus.cc:357] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } }
I20250905 20:00:01.982172 19100 raft_consensus.cc:738] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05ee81c2af44b0ba999a3d8df7bbf97, State: Initialized, Role: FOLLOWER
I20250905 20:00:01.983047 19100 consensus_queue.cc:260] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9f2bc9ce641f42c1b9149e584c915bde" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 37751 } } peers { permanent_uuid: "33c62532982e4fa38fae8e3991477a2d" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 39041 } } peers { permanent_uuid: "a05ee81c2af44b0ba999a3d8df7bbf97" member_type: VOTER last_known_addr { host: "127.14.175.131" port: 35135 } }
I20250905 20:00:01.996436 19100 ts_tablet_manager.cc:1428] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: Time spent starting tablet: real 0.018s user 0.007s sys 0.007s
I20250905 20:00:01.998301 18787 tablet_copy_service.cc:342] P 9f2bc9ce641f42c1b9149e584c915bde: Request end of tablet copy session a05ee81c2af44b0ba999a3d8df7bbf97-aaac0518535247ce87fd064059f1df20 received from {username='slave'} at 127.0.0.1:47024
I20250905 20:00:01.998787 18787 tablet_copy_service.cc:434] P 9f2bc9ce641f42c1b9149e584c915bde: ending tablet copy session a05ee81c2af44b0ba999a3d8df7bbf97-aaac0518535247ce87fd064059f1df20 on tablet aaac0518535247ce87fd064059f1df20 with peer a05ee81c2af44b0ba999a3d8df7bbf97
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
70fe8885f0a948abbdbb57d4c27093fc | 127.14.175.190:38333 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
33c62532982e4fa38fae8e3991477a2d | 127.14.175.130:39041 | HEALTHY | <none> | 0 | 0
9f2bc9ce641f42c1b9149e584c915bde | 127.14.175.129:37751 | HEALTHY | <none> | 1 | 0
a05ee81c2af44b0ba999a3d8df7bbf97 | 127.14.175.131:35135 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet aaac0518535247ce87fd064059f1df20 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
9f2bc9ce641f42c1b9149e584c915bde (127.14.175.129:37751): RUNNING [LEADER]
33c62532982e4fa38fae8e3991477a2d (127.14.175.130:39041): RUNNING
a05ee81c2af44b0ba999a3d8df7bbf97 (127.14.175.131:35135): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 9f2bc9ce641f42c1b9149e584c915bde
B = 33c62532982e4fa38fae8e3991477a2d
C = a05ee81c2af44b0ba999a3d8df7bbf97
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250905 20:00:02.101737 15038 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250905 20:00:02.438968 18926 raft_consensus.cc:1215] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20250905 20:00:02.544991 18842 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:02.550139 18767 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 20:00:02.553397 18916 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
70fe8885f0a948abbdbb57d4c27093fc | 127.14.175.190:38333 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
33c62532982e4fa38fae8e3991477a2d | 127.14.175.130:39041 | HEALTHY | <none> | 0 | 0
9f2bc9ce641f42c1b9149e584c915bde | 127.14.175.129:37751 | HEALTHY | <none> | 1 | 0
a05ee81c2af44b0ba999a3d8df7bbf97 | 127.14.175.131:35135 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250905 20:00:02.748404 15038 log_verifier.cc:126] Checking tablet aaac0518535247ce87fd064059f1df20
I20250905 20:00:02.806658 15038 log_verifier.cc:177] Verified matching terms for 3 ops in tablet aaac0518535247ce87fd064059f1df20
I20250905 20:00:02.810815 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
I20250905 20:00:02.829010 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 20:00:02.829587 15038 tablet_replica.cc:331] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde: stopping tablet replica
I20250905 20:00:02.830107 15038 raft_consensus.cc:2241] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 1 LEADER]: Raft consensus shutting down.
I20250905 20:00:02.831059 15038 raft_consensus.cc:2270] T aaac0518535247ce87fd064059f1df20 P 9f2bc9ce641f42c1b9149e584c915bde [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 20:00:02.850327 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 20:00:02.859655 15038 tablet_server.cc:178] TabletServer@127.14.175.130:0 shutting down...
I20250905 20:00:02.876114 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 20:00:02.876621 15038 tablet_replica.cc:331] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d: stopping tablet replica
I20250905 20:00:02.877076 15038 raft_consensus.cc:2241] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 20:00:02.877481 15038 raft_consensus.cc:2270] T aaac0518535247ce87fd064059f1df20 P 33c62532982e4fa38fae8e3991477a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 20:00:02.896531 15038 tablet_server.cc:195] TabletServer@127.14.175.130:0 shutdown complete.
I20250905 20:00:02.905390 15038 tablet_server.cc:178] TabletServer@127.14.175.131:0 shutting down...
I20250905 20:00:02.923139 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 20:00:02.923689 15038 tablet_replica.cc:331] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97: stopping tablet replica
I20250905 20:00:02.924170 15038 raft_consensus.cc:2241] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 20:00:02.924544 15038 raft_consensus.cc:2270] T aaac0518535247ce87fd064059f1df20 P a05ee81c2af44b0ba999a3d8df7bbf97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 20:00:02.943400 15038 tablet_server.cc:195] TabletServer@127.14.175.131:0 shutdown complete.
I20250905 20:00:02.952566 15038 master.cc:561] Master@127.14.175.190:38333 shutting down...
I20250905 20:00:02.968184 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [term 1 LEADER]: Raft consensus shutting down.
I20250905 20:00:02.968693 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 20:00:02.968955 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P 70fe8885f0a948abbdbb57d4c27093fc: stopping tablet replica
I20250905 20:00:02.986900 15038 master.cc:583] Master@127.14.175.190:38333 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (3780 ms)
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/1
I20250905 20:00:03.013972 15038 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.14.175.190:42647
I20250905 20:00:03.015012 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 20:00:03.019481 19136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 20:00:03.020481 19137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 20:00:03.021878 19139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 20:00:03.023022 15038 server_base.cc:1047] running on GCE node
I20250905 20:00:03.023800 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 20:00:03.023979 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 20:00:03.024128 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102403024108 us; error 0 us; skew 500 ppm
I20250905 20:00:03.024621 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 20:00:03.027029 15038 webserver.cc:480] Webserver started at http://127.14.175.190:33917/ using document root <none> and password file <none>
I20250905 20:00:03.027488 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 20:00:03.027654 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 20:00:03.027910 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 20:00:03.028970 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/master-0-root/instance:
uuid: "3076da69b90c4f8285fb84686bc7ed57"
format_stamp: "Formatted at 2025-09-05 20:00:03 on dist-test-slave-22s0"
I20250905 20:00:03.033411 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20250905 20:00:03.036521 19144 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.037264 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250905 20:00:03.037541 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/master-0-root
uuid: "3076da69b90c4f8285fb84686bc7ed57"
format_stamp: "Formatted at 2025-09-05 20:00:03 on dist-test-slave-22s0"
I20250905 20:00:03.037832 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 20:00:03.055203 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 20:00:03.056191 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 20:00:03.090315 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.190:42647
I20250905 20:00:03.090430 19195 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.190:42647 every 8 connection(s)
I20250905 20:00:03.093905 19196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 20:00:03.104063 19196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57: Bootstrap starting.
I20250905 20:00:03.108289 19196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57: Neither blocks nor log segments found. Creating new log.
I20250905 20:00:03.112167 19196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57: No bootstrap required, opened a new log
I20250905 20:00:03.114068 19196 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3076da69b90c4f8285fb84686bc7ed57" member_type: VOTER }
I20250905 20:00:03.114442 19196 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 20:00:03.114696 19196 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3076da69b90c4f8285fb84686bc7ed57, State: Initialized, Role: FOLLOWER
I20250905 20:00:03.115281 19196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3076da69b90c4f8285fb84686bc7ed57" member_type: VOTER }
I20250905 20:00:03.115727 19196 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 20:00:03.115935 19196 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 20:00:03.116189 19196 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [term 0 FOLLOWER]: Advancing to term 1
I20250905 20:00:03.120805 19196 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3076da69b90c4f8285fb84686bc7ed57" member_type: VOTER }
I20250905 20:00:03.121302 19196 leader_election.cc:304] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3076da69b90c4f8285fb84686bc7ed57; no voters:
I20250905 20:00:03.122398 19196 leader_election.cc:290] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 20:00:03.122756 19199 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 20:00:03.124059 19199 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [term 1 LEADER]: Becoming Leader. State: Replica: 3076da69b90c4f8285fb84686bc7ed57, State: Running, Role: LEADER
I20250905 20:00:03.124712 19199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3076da69b90c4f8285fb84686bc7ed57" member_type: VOTER }
I20250905 20:00:03.125385 19196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 20:00:03.127485 19200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3076da69b90c4f8285fb84686bc7ed57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3076da69b90c4f8285fb84686bc7ed57" member_type: VOTER } }
I20250905 20:00:03.127573 19201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3076da69b90c4f8285fb84686bc7ed57. Latest consensus state: current_term: 1 leader_uuid: "3076da69b90c4f8285fb84686bc7ed57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3076da69b90c4f8285fb84686bc7ed57" member_type: VOTER } }
I20250905 20:00:03.128290 19201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [sys.catalog]: This master's current role is: LEADER
I20250905 20:00:03.128598 19200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [sys.catalog]: This master's current role is: LEADER
I20250905 20:00:03.136544 19206 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 20:00:03.141211 19206 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 20:00:03.142261 15038 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 20:00:03.149278 19206 catalog_manager.cc:1349] Generated new cluster ID: 8d3963b899784a57a742542cd868d932
I20250905 20:00:03.149567 19206 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 20:00:03.170907 19206 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 20:00:03.172163 19206 catalog_manager.cc:1506] Loading token signing keys...
I20250905 20:00:03.190861 19206 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57: Generated new TSK 0
I20250905 20:00:03.191464 19206 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 20:00:03.208801 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 20:00:03.214358 19217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 20:00:03.215855 19218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 20:00:03.216791 19220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 20:00:03.217113 15038 server_base.cc:1047] running on GCE node
I20250905 20:00:03.217896 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 20:00:03.218097 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 20:00:03.218248 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102403218231 us; error 0 us; skew 500 ppm
I20250905 20:00:03.218755 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 20:00:03.220806 15038 webserver.cc:480] Webserver started at http://127.14.175.129:33985/ using document root <none> and password file <none>
I20250905 20:00:03.221268 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 20:00:03.221447 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 20:00:03.221689 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 20:00:03.222777 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-0-root/instance:
uuid: "49e17783ad334b269cb495dd7e3023ab"
format_stamp: "Formatted at 2025-09-05 20:00:03 on dist-test-slave-22s0"
I20250905 20:00:03.226881 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250905 20:00:03.229908 19225 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.230571 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 20:00:03.230826 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-0-root
uuid: "49e17783ad334b269cb495dd7e3023ab"
format_stamp: "Formatted at 2025-09-05 20:00:03 on dist-test-slave-22s0"
I20250905 20:00:03.231103 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 20:00:03.240629 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 20:00:03.241541 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 20:00:03.242908 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 20:00:03.244989 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 20:00:03.245184 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.245406 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 20:00:03.245554 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.282174 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.129:42019
I20250905 20:00:03.282272 19287 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.129:42019 every 8 connection(s)
I20250905 20:00:03.286592 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 20:00:03.293360 19292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 20:00:03.293957 19293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 20:00:03.296748 19295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 20:00:03.297031 15038 server_base.cc:1047] running on GCE node
I20250905 20:00:03.297890 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 20:00:03.298102 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 20:00:03.298106 19288 heartbeater.cc:344] Connected to a master server at 127.14.175.190:42647
I20250905 20:00:03.298559 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102403298538 us; error 0 us; skew 500 ppm
I20250905 20:00:03.298735 19288 heartbeater.cc:461] Registering TS with master...
I20250905 20:00:03.299326 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 20:00:03.299602 19288 heartbeater.cc:507] Master 127.14.175.190:42647 requested a full tablet report, sending...
I20250905 20:00:03.301740 15038 webserver.cc:480] Webserver started at http://127.14.175.130:34297/ using document root <none> and password file <none>
I20250905 20:00:03.301893 19161 ts_manager.cc:194] Registered new tserver with Master: 49e17783ad334b269cb495dd7e3023ab (127.14.175.129:42019)
I20250905 20:00:03.302282 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 20:00:03.302525 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 20:00:03.302809 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 20:00:03.303702 19161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55158
I20250905 20:00:03.303828 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-1-root/instance:
uuid: "c5c20741504e4b7d83bbacf0750977fd"
format_stamp: "Formatted at 2025-09-05 20:00:03 on dist-test-slave-22s0"
I20250905 20:00:03.308533 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 20:00:03.311678 19300 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.312517 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 20:00:03.312762 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-1-root
uuid: "c5c20741504e4b7d83bbacf0750977fd"
format_stamp: "Formatted at 2025-09-05 20:00:03 on dist-test-slave-22s0"
I20250905 20:00:03.313024 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 20:00:03.358711 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 20:00:03.359658 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 20:00:03.361011 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 20:00:03.363277 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 20:00:03.363453 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.363708 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 20:00:03.363847 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.400592 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.130:36289
I20250905 20:00:03.400703 19362 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.130:36289 every 8 connection(s)
I20250905 20:00:03.405303 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 20:00:03.411418 19366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 20:00:03.413265 19367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 20:00:03.416349 19369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 20:00:03.416458 15038 server_base.cc:1047] running on GCE node
I20250905 20:00:03.417171 19363 heartbeater.cc:344] Connected to a master server at 127.14.175.190:42647
I20250905 20:00:03.417451 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250905 20:00:03.417497 19363 heartbeater.cc:461] Registering TS with master...
W20250905 20:00:03.417830 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 20:00:03.418041 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102403418028 us; error 0 us; skew 500 ppm
I20250905 20:00:03.418421 19363 heartbeater.cc:507] Master 127.14.175.190:42647 requested a full tablet report, sending...
I20250905 20:00:03.418618 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 20:00:03.420498 19161 ts_manager.cc:194] Registered new tserver with Master: c5c20741504e4b7d83bbacf0750977fd (127.14.175.130:36289)
I20250905 20:00:03.421830 19161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55168
I20250905 20:00:03.422890 15038 webserver.cc:480] Webserver started at http://127.14.175.131:43497/ using document root <none> and password file <none>
I20250905 20:00:03.423381 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 20:00:03.423554 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 20:00:03.423815 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 20:00:03.424834 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-2-root/instance:
uuid: "29cb8d2a0f2b4225b4137cb62abeed45"
format_stamp: "Formatted at 2025-09-05 20:00:03 on dist-test-slave-22s0"
I20250905 20:00:03.429128 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250905 20:00:03.432353 19374 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.433359 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 20:00:03.433636 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-2-root
uuid: "29cb8d2a0f2b4225b4137cb62abeed45"
format_stamp: "Formatted at 2025-09-05 20:00:03 on dist-test-slave-22s0"
I20250905 20:00:03.433930 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 20:00:03.476387 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 20:00:03.477578 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 20:00:03.478935 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 20:00:03.481079 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 20:00:03.481273 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.481500 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 20:00:03.481653 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.518174 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.131:40629
I20250905 20:00:03.518307 19436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.131:40629 every 8 connection(s)
I20250905 20:00:03.522962 15038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 20:00:03.529893 19440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 20:00:03.533852 19441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 20:00:03.534240 19437 heartbeater.cc:344] Connected to a master server at 127.14.175.190:42647
I20250905 20:00:03.534700 19437 heartbeater.cc:461] Registering TS with master...
I20250905 20:00:03.535567 19437 heartbeater.cc:507] Master 127.14.175.190:42647 requested a full tablet report, sending...
W20250905 20:00:03.536558 19443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 20:00:03.537031 15038 server_base.cc:1047] running on GCE node
I20250905 20:00:03.537869 19161 ts_manager.cc:194] Registered new tserver with Master: 29cb8d2a0f2b4225b4137cb62abeed45 (127.14.175.131:40629)
I20250905 20:00:03.537948 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 20:00:03.538180 15038 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 20:00:03.538313 15038 hybrid_clock.cc:648] HybridClock initialized: now 1757102403538303 us; error 0 us; skew 500 ppm
I20250905 20:00:03.538892 15038 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 20:00:03.539474 19161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55184
I20250905 20:00:03.541512 15038 webserver.cc:480] Webserver started at http://127.14.175.132:41385/ using document root <none> and password file <none>
I20250905 20:00:03.542011 15038 fs_manager.cc:362] Metadata directory not provided
I20250905 20:00:03.542187 15038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 20:00:03.542443 15038 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 20:00:03.543502 15038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-3-root/instance:
uuid: "c39521b5b42342edb84cd4848d93bd18"
format_stamp: "Formatted at 2025-09-05 20:00:03 on dist-test-slave-22s0"
I20250905 20:00:03.548138 15038 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 20:00:03.551318 19448 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.552062 15038 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 20:00:03.552330 15038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-3-root
uuid: "c39521b5b42342edb84cd4848d93bd18"
format_stamp: "Formatted at 2025-09-05 20:00:03 on dist-test-slave-22s0"
I20250905 20:00:03.552592 15038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 20:00:03.592686 15038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 20:00:03.593720 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 20:00:03.595096 15038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 20:00:03.597244 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 20:00:03.597412 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.597683 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 20:00:03.597828 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 20:00:03.635478 15038 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.132:37199
I20250905 20:00:03.635591 19510 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.175.132:37199 every 8 connection(s)
I20250905 20:00:03.648531 19511 heartbeater.cc:344] Connected to a master server at 127.14.175.190:42647
I20250905 20:00:03.648969 19511 heartbeater.cc:461] Registering TS with master...
I20250905 20:00:03.649634 19511 heartbeater.cc:507] Master 127.14.175.190:42647 requested a full tablet report, sending...
I20250905 20:00:03.651453 19161 ts_manager.cc:194] Registered new tserver with Master: c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199)
I20250905 20:00:03.652401 15038 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.013581414s
I20250905 20:00:03.652767 19161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55188
I20250905 20:00:03.671182 15038 test_util.cc:276] Using random seed: 1218106489
I20250905 20:00:03.691433 19161 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55206:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 20:00:03.735453 19328 tablet_service.cc:1468] Processing CreateTablet for tablet 99bd28ce73a64d52a14a23ed33a26124 (DEFAULT_TABLE table=test-workload [id=eed38b5a3b304e49b555e676e64eaee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 20:00:03.736824 19328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99bd28ce73a64d52a14a23ed33a26124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 20:00:03.740144 19253 tablet_service.cc:1468] Processing CreateTablet for tablet 99bd28ce73a64d52a14a23ed33a26124 (DEFAULT_TABLE table=test-workload [id=eed38b5a3b304e49b555e676e64eaee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 20:00:03.740831 19476 tablet_service.cc:1468] Processing CreateTablet for tablet 99bd28ce73a64d52a14a23ed33a26124 (DEFAULT_TABLE table=test-workload [id=eed38b5a3b304e49b555e676e64eaee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 20:00:03.741487 19253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99bd28ce73a64d52a14a23ed33a26124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 20:00:03.742082 19476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99bd28ce73a64d52a14a23ed33a26124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 20:00:03.754680 19534 tablet_bootstrap.cc:492] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd: Bootstrap starting.
I20250905 20:00:03.758662 19535 tablet_bootstrap.cc:492] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab: Bootstrap starting.
I20250905 20:00:03.760430 19536 tablet_bootstrap.cc:492] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18: Bootstrap starting.
I20250905 20:00:03.761915 19534 tablet_bootstrap.cc:654] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd: Neither blocks nor log segments found. Creating new log.
I20250905 20:00:03.764554 19535 tablet_bootstrap.cc:654] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab: Neither blocks nor log segments found. Creating new log.
I20250905 20:00:03.767572 19536 tablet_bootstrap.cc:654] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18: Neither blocks nor log segments found. Creating new log.
I20250905 20:00:03.767702 19534 tablet_bootstrap.cc:492] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd: No bootstrap required, opened a new log
I20250905 20:00:03.768321 19534 ts_tablet_manager.cc:1397] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20250905 20:00:03.769626 19535 tablet_bootstrap.cc:492] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab: No bootstrap required, opened a new log
I20250905 20:00:03.770035 19535 ts_tablet_manager.cc:1397] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20250905 20:00:03.771941 19534 raft_consensus.cc:357] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:03.773020 19534 raft_consensus.cc:383] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 20:00:03.772931 19535 raft_consensus.cc:357] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:03.773654 19534 raft_consensus.cc:738] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c20741504e4b7d83bbacf0750977fd, State: Initialized, Role: FOLLOWER
I20250905 20:00:03.774073 19535 raft_consensus.cc:383] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 20:00:03.774556 19535 raft_consensus.cc:738] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e17783ad334b269cb495dd7e3023ab, State: Initialized, Role: FOLLOWER
I20250905 20:00:03.774523 19534 consensus_queue.cc:260] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:03.775400 19535 consensus_queue.cc:260] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:03.776291 19536 tablet_bootstrap.cc:492] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18: No bootstrap required, opened a new log
I20250905 20:00:03.777235 19363 heartbeater.cc:499] Master 127.14.175.190:42647 was elected leader, sending a full tablet report...
I20250905 20:00:03.777796 19534 ts_tablet_manager.cc:1428] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250905 20:00:03.776847 19536 ts_tablet_manager.cc:1397] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20250905 20:00:03.780601 19536 raft_consensus.cc:357] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:03.781114 19536 raft_consensus.cc:383] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 20:00:03.781349 19536 raft_consensus.cc:738] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39521b5b42342edb84cd4848d93bd18, State: Initialized, Role: FOLLOWER
I20250905 20:00:03.781955 19536 consensus_queue.cc:260] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:03.784396 19535 ts_tablet_manager.cc:1428] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20250905 20:00:03.784754 19511 heartbeater.cc:499] Master 127.14.175.190:42647 was elected leader, sending a full tablet report...
I20250905 20:00:03.785194 19288 heartbeater.cc:499] Master 127.14.175.190:42647 was elected leader, sending a full tablet report...
I20250905 20:00:03.785730 19536 ts_tablet_manager.cc:1428] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250905 20:00:03.924489 19542 raft_consensus.cc:491] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:03.924973 19542 raft_consensus.cc:513] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:03.927066 19542 leader_election.cc:290] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5c20741504e4b7d83bbacf0750977fd (127.14.175.130:36289), 49e17783ad334b269cb495dd7e3023ab (127.14.175.129:42019)
I20250905 20:00:03.936501 19338 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99bd28ce73a64d52a14a23ed33a26124" candidate_uuid: "c39521b5b42342edb84cd4848d93bd18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c20741504e4b7d83bbacf0750977fd" is_pre_election: true
I20250905 20:00:03.936630 19263 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99bd28ce73a64d52a14a23ed33a26124" candidate_uuid: "c39521b5b42342edb84cd4848d93bd18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e17783ad334b269cb495dd7e3023ab" is_pre_election: true
I20250905 20:00:03.937129 19338 raft_consensus.cc:2466] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39521b5b42342edb84cd4848d93bd18 in term 0.
I20250905 20:00:03.937240 19263 raft_consensus.cc:2466] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39521b5b42342edb84cd4848d93bd18 in term 0.
I20250905 20:00:03.938206 19451 leader_election.cc:304] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39521b5b42342edb84cd4848d93bd18, c5c20741504e4b7d83bbacf0750977fd; no voters:
I20250905 20:00:03.938912 19542 raft_consensus.cc:2802] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 20:00:03.939239 19542 raft_consensus.cc:491] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:03.939522 19542 raft_consensus.cc:3058] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 0 FOLLOWER]: Advancing to term 1
I20250905 20:00:03.944202 19542 raft_consensus.cc:513] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:03.945670 19542 leader_election.cc:290] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [CANDIDATE]: Term 1 election: Requested vote from peers c5c20741504e4b7d83bbacf0750977fd (127.14.175.130:36289), 49e17783ad334b269cb495dd7e3023ab (127.14.175.129:42019)
I20250905 20:00:03.946470 19338 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99bd28ce73a64d52a14a23ed33a26124" candidate_uuid: "c39521b5b42342edb84cd4848d93bd18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c20741504e4b7d83bbacf0750977fd"
I20250905 20:00:03.946709 19263 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99bd28ce73a64d52a14a23ed33a26124" candidate_uuid: "c39521b5b42342edb84cd4848d93bd18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e17783ad334b269cb495dd7e3023ab"
I20250905 20:00:03.946997 19338 raft_consensus.cc:3058] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 0 FOLLOWER]: Advancing to term 1
I20250905 20:00:03.947222 19263 raft_consensus.cc:3058] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 0 FOLLOWER]: Advancing to term 1
I20250905 20:00:03.951647 19338 raft_consensus.cc:2466] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39521b5b42342edb84cd4848d93bd18 in term 1.
I20250905 20:00:03.951781 19263 raft_consensus.cc:2466] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39521b5b42342edb84cd4848d93bd18 in term 1.
I20250905 20:00:03.952580 19451 leader_election.cc:304] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39521b5b42342edb84cd4848d93bd18, c5c20741504e4b7d83bbacf0750977fd; no voters:
I20250905 20:00:03.953287 19542 raft_consensus.cc:2802] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 20:00:03.954176 19542 raft_consensus.cc:695] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 1 LEADER]: Becoming Leader. State: Replica: c39521b5b42342edb84cd4848d93bd18, State: Running, Role: LEADER
I20250905 20:00:03.954890 19542 consensus_queue.cc:237] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:03.960726 19159 catalog_manager.cc:5582] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 reported cstate change: term changed from 0 to 1, leader changed from <none> to c39521b5b42342edb84cd4848d93bd18 (127.14.175.132). New cstate: current_term: 1 leader_uuid: "c39521b5b42342edb84cd4848d93bd18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } health_report { overall_health: UNKNOWN } } }
I20250905 20:00:04.211671 19402 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:04.212532 19476 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 20:00:04.212551 19253 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:04.219657 19328 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:04.397845 19542 consensus_queue.cc:1035] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 20:00:04.442140 19542 consensus_queue.cc:1035] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3076da69b90c4f8285fb84686bc7ed57 | 127.14.175.190:42647 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
29cb8d2a0f2b4225b4137cb62abeed45 | 127.14.175.131:40629 | HEALTHY | <none> | 0 | 0
49e17783ad334b269cb495dd7e3023ab | 127.14.175.129:42019 | HEALTHY | <none> | 0 | 0
c39521b5b42342edb84cd4848d93bd18 | 127.14.175.132:37199 | HEALTHY | <none> | 1 | 0
c5c20741504e4b7d83bbacf0750977fd | 127.14.175.130:36289 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 99bd28ce73a64d52a14a23ed33a26124 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
c5c20741504e4b7d83bbacf0750977fd (127.14.175.130:36289): RUNNING
c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199): RUNNING [LEADER]
49e17783ad334b269cb495dd7e3023ab (127.14.175.129:42019): RUNNING
All reported replicas are:
A = c5c20741504e4b7d83bbacf0750977fd
B = c39521b5b42342edb84cd4848d93bd18
C = 49e17783ad334b269cb495dd7e3023ab
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Small | 29cb8d2a0f2b4225b4137cb62abeed45 | 127.14.175.131:40629 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250905 20:00:04.515849 15038 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250905 20:00:04.542048 19437 heartbeater.cc:499] Master 127.14.175.190:42647 was elected leader, sending a full tablet report...
I20250905 20:00:04.934156 19402 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:04.938302 19253 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:04.947185 19476 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 20:00:04.949827 19328 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3076da69b90c4f8285fb84686bc7ed57 | 127.14.175.190:42647 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
29cb8d2a0f2b4225b4137cb62abeed45 | 127.14.175.131:40629 | HEALTHY | <none> | 0 | 0
49e17783ad334b269cb495dd7e3023ab | 127.14.175.129:42019 | HEALTHY | <none> | 0 | 0
c39521b5b42342edb84cd4848d93bd18 | 127.14.175.132:37199 | HEALTHY | <none> | 1 | 0
c5c20741504e4b7d83bbacf0750977fd | 127.14.175.130:36289 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Small | 29cb8d2a0f2b4225b4137cb62abeed45 | 127.14.175.131:40629 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250905 20:00:05.187378 15038 log_verifier.cc:126] Checking tablet 99bd28ce73a64d52a14a23ed33a26124
I20250905 20:00:05.244657 15038 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 99bd28ce73a64d52a14a23ed33a26124
I20250905 20:00:05.245222 15038 tablet_replica.cc:331] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18: stopping tablet replica
I20250905 20:00:05.245790 15038 raft_consensus.cc:2241] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 1 LEADER]: Raft consensus shutting down.
I20250905 20:00:05.246757 15038 raft_consensus.cc:2270] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 20:00:05.465761 19402 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:05.478260 19328 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:05.478752 19476 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:05.480324 19253 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3076da69b90c4f8285fb84686bc7ed57 | 127.14.175.190:42647 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
29cb8d2a0f2b4225b4137cb62abeed45 | 127.14.175.131:40629 | HEALTHY | <none> | 0 | 0
49e17783ad334b269cb495dd7e3023ab | 127.14.175.129:42019 | HEALTHY | <none> | 0 | 0
c39521b5b42342edb84cd4848d93bd18 | 127.14.175.132:37199 | HEALTHY | <none> | 0 | 0
c5c20741504e4b7d83bbacf0750977fd | 127.14.175.130:36289 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 99bd28ce73a64d52a14a23ed33a26124 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
c5c20741504e4b7d83bbacf0750977fd (127.14.175.130:36289): RUNNING
c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
49e17783ad334b269cb495dd7e3023ab (127.14.175.129:42019): RUNNING
All reported replicas are:
A = c5c20741504e4b7d83bbacf0750977fd
B = c39521b5b42342edb84cd4848d93bd18
C = 49e17783ad334b269cb495dd7e3023ab
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Small | 29cb8d2a0f2b4225b4137cb62abeed45 | 127.14.175.131:40629 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250905 20:00:05.701917 15038 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250905 20:00:06.063983 19402 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:06.073132 19253 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:06.079536 19476 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:06.084792 19328 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3076da69b90c4f8285fb84686bc7ed57 | 127.14.175.190:42647 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
29cb8d2a0f2b4225b4137cb62abeed45 | 127.14.175.131:40629 | HEALTHY | <none> | 0 | 0
49e17783ad334b269cb495dd7e3023ab | 127.14.175.129:42019 | HEALTHY | <none> | 0 | 0
c39521b5b42342edb84cd4848d93bd18 | 127.14.175.132:37199 | HEALTHY | <none> | 0 | 0
c5c20741504e4b7d83bbacf0750977fd | 127.14.175.130:36289 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 99bd28ce73a64d52a14a23ed33a26124 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
c5c20741504e4b7d83bbacf0750977fd (127.14.175.130:36289): RUNNING
c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
49e17783ad334b269cb495dd7e3023ab (127.14.175.129:42019): RUNNING
All reported replicas are:
A = c5c20741504e4b7d83bbacf0750977fd
B = c39521b5b42342edb84cd4848d93bd18
C = 49e17783ad334b269cb495dd7e3023ab
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Small | 29cb8d2a0f2b4225b4137cb62abeed45 | 127.14.175.131:40629 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250905 20:00:06.288954 15038 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250905 20:00:06.466400 19673 raft_consensus.cc:491] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c39521b5b42342edb84cd4848d93bd18)
I20250905 20:00:06.467038 19673 raft_consensus.cc:513] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:06.469841 19673 leader_election.cc:290] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5c20741504e4b7d83bbacf0750977fd (127.14.175.130:36289), c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199)
I20250905 20:00:06.480912 19338 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99bd28ce73a64d52a14a23ed33a26124" candidate_uuid: "49e17783ad334b269cb495dd7e3023ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5c20741504e4b7d83bbacf0750977fd" is_pre_election: true
I20250905 20:00:06.482586 19486 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99bd28ce73a64d52a14a23ed33a26124" candidate_uuid: "49e17783ad334b269cb495dd7e3023ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c39521b5b42342edb84cd4848d93bd18" is_pre_election: true
W20250905 20:00:06.483807 19226 leader_election.cc:343] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199): Illegal state: cannot vote while shut down
I20250905 20:00:06.484179 19226 leader_election.cc:304] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49e17783ad334b269cb495dd7e3023ab; no voters: c39521b5b42342edb84cd4848d93bd18, c5c20741504e4b7d83bbacf0750977fd
I20250905 20:00:06.484797 19673 raft_consensus.cc:2747] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 20:00:06.495816 19676 raft_consensus.cc:491] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c39521b5b42342edb84cd4848d93bd18)
I20250905 20:00:06.496457 19676 raft_consensus.cc:513] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:06.499217 19676 leader_election.cc:290] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199), 49e17783ad334b269cb495dd7e3023ab (127.14.175.129:42019)
I20250905 20:00:06.511535 19263 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99bd28ce73a64d52a14a23ed33a26124" candidate_uuid: "c5c20741504e4b7d83bbacf0750977fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49e17783ad334b269cb495dd7e3023ab" is_pre_election: true
I20250905 20:00:06.512156 19263 raft_consensus.cc:2466] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5c20741504e4b7d83bbacf0750977fd in term 1.
I20250905 20:00:06.512034 19486 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99bd28ce73a64d52a14a23ed33a26124" candidate_uuid: "c5c20741504e4b7d83bbacf0750977fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c39521b5b42342edb84cd4848d93bd18" is_pre_election: true
I20250905 20:00:06.513386 19302 leader_election.cc:304] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e17783ad334b269cb495dd7e3023ab, c5c20741504e4b7d83bbacf0750977fd; no voters:
W20250905 20:00:06.514025 19301 leader_election.cc:343] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199): Illegal state: cannot vote while shut down
I20250905 20:00:06.514058 19676 raft_consensus.cc:2802] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 20:00:06.514766 19676 raft_consensus.cc:491] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 1 FOLLOWER]: Starting leader election (detected failure of leader c39521b5b42342edb84cd4848d93bd18)
I20250905 20:00:06.515117 19676 raft_consensus.cc:3058] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 1 FOLLOWER]: Advancing to term 2
I20250905 20:00:06.523437 19676 raft_consensus.cc:513] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:06.525691 19676 leader_election.cc:290] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [CANDIDATE]: Term 2 election: Requested vote from peers c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199), 49e17783ad334b269cb495dd7e3023ab (127.14.175.129:42019)
I20250905 20:00:06.526405 19486 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99bd28ce73a64d52a14a23ed33a26124" candidate_uuid: "c5c20741504e4b7d83bbacf0750977fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c39521b5b42342edb84cd4848d93bd18"
I20250905 20:00:06.527631 19263 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99bd28ce73a64d52a14a23ed33a26124" candidate_uuid: "c5c20741504e4b7d83bbacf0750977fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49e17783ad334b269cb495dd7e3023ab"
W20250905 20:00:06.527832 19301 leader_election.cc:343] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199): Illegal state: cannot vote while shut down
I20250905 20:00:06.528082 19263 raft_consensus.cc:3058] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 1 FOLLOWER]: Advancing to term 2
I20250905 20:00:06.534756 19263 raft_consensus.cc:2466] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5c20741504e4b7d83bbacf0750977fd in term 2.
I20250905 20:00:06.535625 19302 leader_election.cc:304] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e17783ad334b269cb495dd7e3023ab, c5c20741504e4b7d83bbacf0750977fd; no voters: c39521b5b42342edb84cd4848d93bd18
I20250905 20:00:06.537101 19676 raft_consensus.cc:2802] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 2 FOLLOWER]: Leader election won for term 2
I20250905 20:00:06.538635 19676 raft_consensus.cc:695] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 2 LEADER]: Becoming Leader. State: Replica: c5c20741504e4b7d83bbacf0750977fd, State: Running, Role: LEADER
I20250905 20:00:06.539528 19676 consensus_queue.cc:237] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:06.546121 19161 catalog_manager.cc:5582] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd reported cstate change: term changed from 1 to 2, leader changed from c39521b5b42342edb84cd4848d93bd18 (127.14.175.132) to c5c20741504e4b7d83bbacf0750977fd (127.14.175.130). New cstate: current_term: 2 leader_uuid: "c5c20741504e4b7d83bbacf0750977fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39521b5b42342edb84cd4848d93bd18" member_type: VOTER last_known_addr { host: "127.14.175.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } health_report { overall_health: UNKNOWN } } }
I20250905 20:00:06.733492 19402 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:06.739041 19253 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:06.744668 19476 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:06.748221 19328 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3076da69b90c4f8285fb84686bc7ed57 | 127.14.175.190:42647 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
29cb8d2a0f2b4225b4137cb62abeed45 | 127.14.175.131:40629 | HEALTHY | <none> | 0 | 0
49e17783ad334b269cb495dd7e3023ab | 127.14.175.129:42019 | HEALTHY | <none> | 0 | 0
c39521b5b42342edb84cd4848d93bd18 | 127.14.175.132:37199 | HEALTHY | <none> | 0 | 0
c5c20741504e4b7d83bbacf0750977fd | 127.14.175.130:36289 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 99bd28ce73a64d52a14a23ed33a26124 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
c5c20741504e4b7d83bbacf0750977fd (127.14.175.130:36289): RUNNING [LEADER]
c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
49e17783ad334b269cb495dd7e3023ab (127.14.175.129:42019): RUNNING
All reported replicas are:
A = c5c20741504e4b7d83bbacf0750977fd
B = c39521b5b42342edb84cd4848d93bd18
C = 49e17783ad334b269cb495dd7e3023ab
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Small | 29cb8d2a0f2b4225b4137cb62abeed45 | 127.14.175.131:40629 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250905 20:00:06.983572 15038 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250905 20:00:07.012454 19263 raft_consensus.cc:1273] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 2 FOLLOWER]: Refusing update from remote peer c5c20741504e4b7d83bbacf0750977fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 20:00:07.013711 19676 consensus_queue.cc:1035] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 20:00:07.022128 19301 consensus_queue.cc:579] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [LEADER]: The tablet replica hosted on peer c39521b5b42342edb84cd4848d93bd18 has failed
W20250905 20:00:07.022716 19301 consensus_peers.cc:489] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd -> Peer c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199): Couldn't send request to peer c39521b5b42342edb84cd4848d93bd18. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20250905 20:00:07.030262 19338 consensus_queue.cc:237] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } }
I20250905 20:00:07.033943 19263 raft_consensus.cc:1273] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 2 FOLLOWER]: Refusing update from remote peer c5c20741504e4b7d83bbacf0750977fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 20:00:07.034984 19686 consensus_queue.cc:1035] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 20:00:07.040421 19676 raft_consensus.cc:2953] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER c39521b5b42342edb84cd4848d93bd18 (127.14.175.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } } }
I20250905 20:00:07.041850 19263 raft_consensus.cc:2953] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER c39521b5b42342edb84cd4848d93bd18 (127.14.175.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } } }
I20250905 20:00:07.049347 19147 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 99bd28ce73a64d52a14a23ed33a26124 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 20:00:07.052400 19161 catalog_manager.cc:5582] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd reported cstate change: config changed from index -1 to 3, VOTER c39521b5b42342edb84cd4848d93bd18 (127.14.175.132) evicted. New cstate: current_term: 2 leader_uuid: "c5c20741504e4b7d83bbacf0750977fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } health_report { overall_health: HEALTHY } } }
I20250905 20:00:07.062628 19476 tablet_service.cc:1515] Processing DeleteTablet for tablet 99bd28ce73a64d52a14a23ed33a26124 with delete_type TABLET_DATA_TOMBSTONED (TS c39521b5b42342edb84cd4848d93bd18 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57190
I20250905 20:00:07.064838 19717 ts_tablet_manager.cc:1905] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 20:00:07.066149 19338 consensus_queue.cc:237] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } } peers { permanent_uuid: "29cb8d2a0f2b4225b4137cb62abeed45" member_type: NON_VOTER last_known_addr { host: "127.14.175.131" port: 40629 } attrs { promote: true } }
I20250905 20:00:07.073320 19263 raft_consensus.cc:1273] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 2 FOLLOWER]: Refusing update from remote peer c5c20741504e4b7d83bbacf0750977fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250905 20:00:07.074805 19676 consensus_queue.cc:1035] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 20:00:07.081319 19686 raft_consensus.cc:2953] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER 29cb8d2a0f2b4225b4137cb62abeed45 (127.14.175.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } } peers { permanent_uuid: "29cb8d2a0f2b4225b4137cb62abeed45" member_type: NON_VOTER last_known_addr { host: "127.14.175.131" port: 40629 } attrs { promote: true } } }
W20250905 20:00:07.082856 19303 consensus_peers.cc:489] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd -> Peer 29cb8d2a0f2b4225b4137cb62abeed45 (127.14.175.131:40629): Couldn't send request to peer 29cb8d2a0f2b4225b4137cb62abeed45. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99bd28ce73a64d52a14a23ed33a26124. This is attempt 1: this message will repeat every 5th retry.
I20250905 20:00:07.083310 19717 ts_tablet_manager.cc:1918] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250905 20:00:07.083739 19717 log.cc:1199] T 99bd28ce73a64d52a14a23ed33a26124 P c39521b5b42342edb84cd4848d93bd18: Deleting WAL directory at /tmp/dist-test-taskwnaFL9/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1757102344413625-15038-0/minicluster-data/ts-3-root/wals/99bd28ce73a64d52a14a23ed33a26124
I20250905 20:00:07.083014 19262 raft_consensus.cc:2953] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER 29cb8d2a0f2b4225b4137cb62abeed45 (127.14.175.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } } peers { permanent_uuid: "29cb8d2a0f2b4225b4137cb62abeed45" member_type: NON_VOTER last_known_addr { host: "127.14.175.131" port: 40629 } attrs { promote: true } } }
I20250905 20:00:07.085621 19145 catalog_manager.cc:4928] TS c39521b5b42342edb84cd4848d93bd18 (127.14.175.132:37199): tablet 99bd28ce73a64d52a14a23ed33a26124 (table test-workload [id=eed38b5a3b304e49b555e676e64eaee8]) successfully deleted
I20250905 20:00:07.088845 19147 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 99bd28ce73a64d52a14a23ed33a26124 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250905 20:00:07.092213 19161 catalog_manager.cc:5582] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd reported cstate change: config changed from index 3 to 4, NON_VOTER 29cb8d2a0f2b4225b4137cb62abeed45 (127.14.175.131) added. New cstate: current_term: 2 leader_uuid: "c5c20741504e4b7d83bbacf0750977fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29cb8d2a0f2b4225b4137cb62abeed45" member_type: NON_VOTER last_known_addr { host: "127.14.175.131" port: 40629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 20:00:07.534608 19402 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:07.546159 19253 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:07.547271 19328 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 20:00:07.550671 19476 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 20:00:07.635665 19747 ts_tablet_manager.cc:927] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45: Initiating tablet copy from peer c5c20741504e4b7d83bbacf0750977fd (127.14.175.130:36289)
I20250905 20:00:07.673820 19747 tablet_copy_client.cc:323] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45: tablet copy: Beginning tablet copy session from remote peer at address 127.14.175.130:36289
I20250905 20:00:07.714995 19348 tablet_copy_service.cc:140] P c5c20741504e4b7d83bbacf0750977fd: Received BeginTabletCopySession request for tablet 99bd28ce73a64d52a14a23ed33a26124 from peer 29cb8d2a0f2b4225b4137cb62abeed45 ({username='slave'} at 127.0.0.1:59086)
I20250905 20:00:07.715762 19348 tablet_copy_service.cc:161] P c5c20741504e4b7d83bbacf0750977fd: Beginning new tablet copy session on tablet 99bd28ce73a64d52a14a23ed33a26124 from peer 29cb8d2a0f2b4225b4137cb62abeed45 at {username='slave'} at 127.0.0.1:59086: session id = 29cb8d2a0f2b4225b4137cb62abeed45-99bd28ce73a64d52a14a23ed33a26124
I20250905 20:00:07.725234 19348 tablet_copy_source_session.cc:215] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 20:00:07.735868 19747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99bd28ce73a64d52a14a23ed33a26124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 20:00:07.754383 19747 tablet_copy_client.cc:806] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45: tablet copy: Starting download of 0 data blocks...
I20250905 20:00:07.755056 19747 tablet_copy_client.cc:670] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45: tablet copy: Starting download of 1 WAL segments...
I20250905 20:00:07.759042 19747 tablet_copy_client.cc:538] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 20:00:07.767478 19747 tablet_bootstrap.cc:492] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45: Bootstrap starting.
I20250905 20:00:07.786204 19747 tablet_bootstrap.cc:492] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 20:00:07.787098 19747 tablet_bootstrap.cc:492] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45: Bootstrap complete.
I20250905 20:00:07.787644 19747 ts_tablet_manager.cc:1397] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20250905 20:00:07.789698 19747 raft_consensus.cc:357] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } } peers { permanent_uuid: "29cb8d2a0f2b4225b4137cb62abeed45" member_type: NON_VOTER last_known_addr { host: "127.14.175.131" port: 40629 } attrs { promote: true } }
I20250905 20:00:07.790277 19747 raft_consensus.cc:738] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 29cb8d2a0f2b4225b4137cb62abeed45, State: Initialized, Role: LEARNER
I20250905 20:00:07.790768 19747 consensus_queue.cc:260] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5c20741504e4b7d83bbacf0750977fd" member_type: VOTER last_known_addr { host: "127.14.175.130" port: 36289 } } peers { permanent_uuid: "49e17783ad334b269cb495dd7e3023ab" member_type: VOTER last_known_addr { host: "127.14.175.129" port: 42019 } } peers { permanent_uuid: "29cb8d2a0f2b4225b4137cb62abeed45" member_type: NON_VOTER last_known_addr { host: "127.14.175.131" port: 40629 } attrs { promote: true } }
I20250905 20:00:07.793089 19747 ts_tablet_manager.cc:1428] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20250905 20:00:07.794798 19348 tablet_copy_service.cc:342] P c5c20741504e4b7d83bbacf0750977fd: Request end of tablet copy session 29cb8d2a0f2b4225b4137cb62abeed45-99bd28ce73a64d52a14a23ed33a26124 received from {username='slave'} at 127.0.0.1:59086
I20250905 20:00:07.795212 19348 tablet_copy_service.cc:434] P c5c20741504e4b7d83bbacf0750977fd: ending tablet copy session 29cb8d2a0f2b4225b4137cb62abeed45-99bd28ce73a64d52a14a23ed33a26124 on tablet 99bd28ce73a64d52a14a23ed33a26124 with peer 29cb8d2a0f2b4225b4137cb62abeed45
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3076da69b90c4f8285fb84686bc7ed57 | 127.14.175.190:42647 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
29cb8d2a0f2b4225b4137cb62abeed45 | 127.14.175.131:40629 | HEALTHY | <none> | 0 | 0
49e17783ad334b269cb495dd7e3023ab | 127.14.175.129:42019 | HEALTHY | <none> | 0 | 0
c39521b5b42342edb84cd4848d93bd18 | 127.14.175.132:37199 | HEALTHY | <none> | 0 | 0
c5c20741504e4b7d83bbacf0750977fd | 127.14.175.130:36289 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Small | c39521b5b42342edb84cd4848d93bd18 | 127.14.175.132:37199 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250905 20:00:07.824186 15038 log_verifier.cc:126] Checking tablet 99bd28ce73a64d52a14a23ed33a26124
I20250905 20:00:07.881731 15038 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 99bd28ce73a64d52a14a23ed33a26124
I20250905 20:00:07.884974 15038 tablet_server.cc:178] TabletServer@127.14.175.129:0 shutting down...
I20250905 20:00:07.911880 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 20:00:07.912562 15038 tablet_replica.cc:331] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab: stopping tablet replica
I20250905 20:00:07.913074 15038 raft_consensus.cc:2241] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 20:00:07.913586 15038 raft_consensus.cc:2270] T 99bd28ce73a64d52a14a23ed33a26124 P 49e17783ad334b269cb495dd7e3023ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 20:00:07.934429 15038 tablet_server.cc:195] TabletServer@127.14.175.129:0 shutdown complete.
I20250905 20:00:07.945314 15038 tablet_server.cc:178] TabletServer@127.14.175.130:0 shutting down...
I20250905 20:00:07.964583 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 20:00:07.965293 15038 tablet_replica.cc:331] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd: stopping tablet replica
I20250905 20:00:07.965857 15038 raft_consensus.cc:2241] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 2 LEADER]: Raft consensus shutting down.
I20250905 20:00:07.966740 15038 raft_consensus.cc:2270] T 99bd28ce73a64d52a14a23ed33a26124 P c5c20741504e4b7d83bbacf0750977fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 20:00:07.986833 15038 tablet_server.cc:195] TabletServer@127.14.175.130:0 shutdown complete.
I20250905 20:00:07.997210 15038 tablet_server.cc:178] TabletServer@127.14.175.131:0 shutting down...
I20250905 20:00:08.017778 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 20:00:08.018651 15038 tablet_replica.cc:331] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45: stopping tablet replica
I20250905 20:00:08.019225 15038 raft_consensus.cc:2241] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45 [term 2 LEARNER]: Raft consensus shutting down.
I20250905 20:00:08.019608 15038 raft_consensus.cc:2270] T 99bd28ce73a64d52a14a23ed33a26124 P 29cb8d2a0f2b4225b4137cb62abeed45 [term 2 LEARNER]: Raft consensus is shut down!
I20250905 20:00:08.038587 15038 tablet_server.cc:195] TabletServer@127.14.175.131:0 shutdown complete.
I20250905 20:00:08.048782 15038 tablet_server.cc:178] TabletServer@127.14.175.132:0 shutting down...
I20250905 20:00:08.069541 15038 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 20:00:08.086134 15038 tablet_server.cc:195] TabletServer@127.14.175.132:0 shutdown complete.
I20250905 20:00:08.096307 15038 master.cc:561] Master@127.14.175.190:42647 shutting down...
I20250905 20:00:08.116014 15038 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [term 1 LEADER]: Raft consensus shutting down.
I20250905 20:00:08.116505 15038 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 20:00:08.116784 15038 tablet_replica.cc:331] T 00000000000000000000000000000000 P 3076da69b90c4f8285fb84686bc7ed57: stopping tablet replica
I20250905 20:00:08.133788 15038 master.cc:583] Master@127.14.175.190:42647 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (5149 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (8930 ms total)
[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (63586 ms total)
[ PASSED ] 11 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[ FAILED ] 1 test, listed below:
[ FAILED ] TsTabletManagerITest.TestTableStats
1 FAILED TEST
I20250905 20:00:08.171125 15038 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~9 seconds ago
I20250905 20:00:08.171435 15038 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 7 messages since previous log ~13 seconds ago
I20250905 20:00:08.171586 15038 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 104 messages since previous log ~55 seconds ago
I20250905 20:00:08.171825 15038 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 7 messages since previous log ~12 seconds ago